编译

1. requirement
   安装 protobuf，
   经测试2.6版本的可以，3.0暂时还没有正式版。 可以修改project/pom.xml里面的protobuf.version 由2.5.0改成2.6.0，
   于3.0暂时还没有正式版， 所以暂时还不行， 会报错protoc version is ‘libprotoc 3.0.0’, expected version is ‘2.5.0’
   mvn package -Pdist -Ptar -Pdocs -skipTests

   1 2 3 4 5 6 7

   cd hadoop-mapreduce-project export Platform=x64 mvn clean install assembly:assembly -Pnative ## Issue the following command instead (while still being in the hadoop-mapreduce-project folder: mvn package -Pdist,native -DskipTests=true -Dtar cd .. # move back to top-level folder mvn package -Pdist,native -DskipTests=true -Dtar

编译中可能出现的错误

1
2
3
4

Error: Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.2.1:assembly (default-cli) on project hadoop-mapreduce: Error reading assemblies: No assembly descriptors found. -> [Help 1]

The C/C++ header for freetype (ft2build.h)  could not be found.  You may need to install the
apt-get install freetype

编译指令
mvn clean package -Pdist -Dtar -Dmaven.javadoc.skip=true -DskipTests -fail-at-end -Pnative

参考

geting-started-with-hadoop-2-2-0-building

PECS原则：生产者（Producer）使用extends，消费者（Consumer）使用super。

•生产者使用extends

如果你需要一个列表提供T类型的元素（即你想从列表中读取T类型的元素），你需要把这个列表声明成<? extends T>，比如List<? extends Integer>，因此你不能往该列表中添加任何元素。
•消费者使用super

如果需要一个列表使用T类型的元素（即你想把T类型的元素加入到列表中），你需要把这个列表声明成<? super T>，比如List<? super Integer>，因此你不能保证从中读取到的元素的类型。
•即是生产者，也是消费者

如果一个列表即要生产，又要消费，你不能使用泛型通配符声明列表，比如List。
请参考java.util.Collections里的copy方法(JDK1.7)：
来自stackoverflow的例子

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80

PECS (short for "Producer extends and Consumer super") can be explained by : Get and Put Principle
Get And Put Principle (From Java Generics and Collections)

It states,

    use an extends wildcard when you only get values out of a structure
    use a super wildcard when you only put values into a structure
    and don’t use a wildcard when you both get and put.

Let's understand it by example:

1. For Extends Wildcard(get values i.e Producer extends)

Here is a method, that takes a collection of numbers, converts each to a double, and sums them up

public static double sum(Collection<? extends Number> nums) {
   double s = 0.0;
   for (Number num : nums) 
      s += num.doubleValue();
   return s;
}

Let's call the method :

List<Integer>ints = Arrays.asList(1,2,3);
assert sum(ints) == 6.0;
List<Double>doubles = Arrays.asList(2.78,3.14);
assert sum(doubles) == 5.92;
List<Number>nums = Arrays.<Number>asList(1,2,2.78,3.14);
assert sum(nums) == 8.92;

Since, sum() method uses extends, all of the following calls are legal. The first two calls would not be legal if extends was not used.

EXCEPTION : You cannot put anything into a type declared with an extends wildcard—except for the value null, which belongs to every reference type:

List<Integer> ints = new ArrayList<Integer>();
ints.add(1);
ints.add(2);
List<? extends Number> nums = ints;
nums.add(null);  // ok
assert nums.toString().equals("[1, 2, null]");

2. For Super Wildcard(put values i.e Consumer super)

Here is a method, that takes a collection of numbers and an int n, and puts the first n integers, starting from zero, into the collection:

public static void count(Collection<? super Integer> ints, int n) {
    for (int i = 0; i < n; i++) ints.add(i);
}

Let's call the method :

List<Integer>ints = new ArrayList<Integer>();
count(ints, 5);
assert ints.toString().equals("[0, 1, 2, 3, 4]");
List<Number>nums = new ArrayList<Number>();
count(nums, 5); nums.add(5.0);
assert nums.toString().equals("[0, 1, 2, 3, 4, 5.0]");
List<Object>objs = new ArrayList<Object>();
count(objs, 5); objs.add("five");
assert objs.toString().equals("[0, 1, 2, 3, 4, five]");

Since, count() method uses super, all of the following calls are legal: The last two calls would not be legal if super was not used.

EXCEPTION : you cannot get anything out from a type declared with a super wildcard—except for a value of type Object, which is a supertype of every reference type:

List<Object> objs = Arrays.<Object>asList(1,"two");
List<? super Integer> ints = objs;
String str = "";
for (Object obj : ints) str += obj.toString();
assert str.equals("1two");

3. When both Get and Put, don't Use wildcard

Whenever you both put values into and get values out of the same structure, you should not use a wildcard.

public static double sumCount(Collection<Number> nums, int n) {
   count(nums, n);
   return sum(nums);
}

“PECS” is from the collection’s point of view. If you are only pulling items from a generic collection, it is a producer and you should use extends; if you are only stuffing items in, it is a consumer and you should use super. If you do both with the same collection, you shouldn’t use either extends or super.

Suppose you have a method that takes as its parameter a collection of things, but you want it to be more flexible than just accepting a Collection.

Case 1: You want to go through the collection and do things with each item.
Then the list is a producer, so you should use a Collection<? extends Thing>.

The reasoning is that a Collection<? extends Thing> could hold any subtype of Thing, and thus each element will behave as a Thing when you perform your operation. (You actually cannot add anything to a Collection<? extends Thing>, because you cannot know at runtime which specific subtype of Thing the collection holds.)

Case 2: You want to add things to the collection.
Then the list is a consumer, so you should use a Collection<? super Thing>.

The reasoning here is that unlike Collection<? extends Thing>, Collection<? super Thing> can always hold a Thing no matter what the actual parameterized type is. Here you don’t care what is already in the list as long as it will allow a Thing to be added; this is what ? super Thing guarantees.

I’m always trying to think about it this way: A producer is allowed to produce something more specific, hence extends, a consumer is allowed to accept something more general, hence super. – Feuermurmel May 7 ‘13 at 13:11

Another way to remember the producer/consumer distinction is to think of a method signature. If you have a method doSomethingWithList(List list), you are consuming the list and so will need covariance / extends (or an invariant List). On the other hand if your method is List doSomethingProvidingList, then you are producing the List and will need contravariance / super (or an invariant List). – Raman Jan 24 ‘14 at 19:20

matplotlib

windows下安装pip install matplotlib， 需要安装freetype, png库
从源码安装的话报错 ： The following required packages can not be built: freetype, png
依赖包：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

               gtk3agg: no  [Requires pygobject to be installed.]
             gtk3cairo: no  [Requires cairocffi or pycairo to be installed.]
                gtkagg: no  [Requires pygtk]
                 tkagg: no  [The C/C++ header for Tk (tk.h) could not be
                        found.  You may need to install the development
                        package.]
                 wxagg: no  [requires wxPython]
                   gtk: no  [Requires pygtk]
                   agg: yes [installing]
                 cairo: no  [cairocffi or pycairo not found]
             windowing: yes [installing, installing]

OPTIONAL LATEX DEPENDENCIES
                dvipng: no
           ghostscript: no
                 latex: no
               pdftops: no

OPTIONAL PACKAGE DATA
                  dlls: no  [skipping due to configuration]

测试工具

使用wget:

1
2
3
4
5
6
7

# mirror: 镜像  adjust-extension
wget --mirror --cvonvert-links --adjust-extension --page-requistites --no-parent -nc http://xxxxxx
#简写形式
wget -mkEpnp -nc 
wget -m -np -p http://xxxx
···
pavuk

pavuk -enable_js -fnrules F ‘.php?‘ ‘%o.php’ -tr_str_str ‘?’ ‘questionmark‘ -norobots -dont_limit_inlines -dont_leave_dir
http://xxxxx > OUT 2>ERR
```