-
Notifications
You must be signed in to change notification settings - Fork 3
/
Copy pathREADME.Linden
960 lines (784 loc) · 43.6 KB
/
README.Linden
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
0. Pre-Checkin Checklist
Performed from top of repo, default branch, head:
[ ] Is tag 'current' at or near head of 'vendor'?
hg heads
hg tags
[ ] Expected differences between vendor and default? Very
little of the original source should be modified.
hg diff -rcurrent curl
[ ] Are the 'vendor' and 'default' branch source directories
'curl' and not 'curl-<version>'?
[ ] Is curlbuild.h identical to distribution?
hg diff -rcurrent curl/include/curl/curlbuild.h
[ ] Is Makefile identical to distribution? Not strictly
required but a good idea.
hg diff -rcurrent curl/Makefile
[ ] Examine the build logs to make certain that the data type
primitives and their formats are in agreement. Good
example from a 32-bit Linux build:
checking size of long... 4
checking size of void*... 4
checking for 64-bit curl_off_t data type... int64_t
checking size of curl_off_t... 8
checking formatting string directive for curl_off_t... "lld"
checking formatting string directive for unsigned curl_off_t... "llu"
checking constant suffix string for curl_off_t... LL
checking constant suffix string for unsigned curl_off_t... ULL
[ ] The curl configuration procedure has trouble locating custom
versions of support libraries (zlib, ssl, ssh2, etc.). So
examing the build log and verify that the right libraries are
being picked up. Example from Linux:
checking for OpenSSL headers version... 1.0.1 - 0x1000107fL
checking for OpenSSL library version... 1.0.1
checking for OpenSSL headers and library versions matching... yes
Example from Linux of zlib detection:
checking zlib.h usability... yes
checking zlib.h presence... yes
checking for zlib.h... yes
checking for gzread in -lz... yes
configure: found both libz and libz.h header
During development, you might find it useful to rename any
system version of the zlib.h header so that it isn't picked
up accidentally.
In addition to the 'curl --version' command, you can look
for version strings in the 'curl' program itself:
$ strings stage/bin/curl | grep 1\\.[0-9]\\.
--dns-servers DNS server addrs to use: 1.1.1.1;2.2.2.2
1.2.8
1.2.0.4
SSLv2 part of OpenSSL 1.0.1g 7 Apr 2014
SSLv3 part of OpenSSL 1.0.1g 7 Apr 2014
...
\CAST part of OpenSSL 1.0.1g 7 Apr 2014
deflate 1.2.8 Copyright 1995-2013 Jean-loup Gailly and Mark Adler
inflate 1.2.8 Copyright 1995-2013 Mark Adler
With the '1.2.8' indicating the targeted libz (1.2.3.4
typically means a platform libz).
Static archives will show even less version information:
$ strings stage/lib/release/libcurl.a | grep 1\\.[0-9]\\.
1.2.8
1.2.0.4
[ ] Examine build logs or run the built 'curl' executable to
verify function. The '--version' option will give useful
library information:
$ stage/bin/curl --version
curl 7.42.0 (x86_64-unknown-linux-gnu) libcurl/7.47.0 OpenSSL/1.0.1g zlib/1.2.8
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP
Run a few curl commands to a website. You may need to copy
DLLs, create a Resources peer directory with symlinks to
.dylibs or set LD_LIBRARY_PATH to locate OpenSSL shared
libraries at runtime. (build-cmd.sh does this for its own
operations.)
Confirm a fetch and the progress display works as expected:
$ stage/bin/curl -o /dev/null http://example.com/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1270 100 1270 0 0 55548 0 --:--:-- --:--:-- --:--:-- 57727
[ ] On Mac, unit tests are disabled by default because they invoke
a lot of network activity that triggers the OS X firewall.
Inhibit the disable ('export DISABLE_UNIT_TESTS=0') and run
builds on your machine to exercise the unit tests. There may
be some flakiness in the tests but they have been useful.
[ ] Are you able to revert the special pipelining changes for curl
bugs 1420 (https://sourceforge.net/p/curl/bugs/1420/)? This
is done by changesets a8784f6e153f and 46fd1a2376ab.
1. Introduction
Build of curl library from curl.haxx.se.
Repo structure *now* mostly follows standard conventions (see
section at end).
Prior to 7.24.0, repo didn't follow standard convention and each
release was disconnected from previous releases. Modifications
to branch 'default' were re-implemented on every update.
Running 'configure' modifies source files. Don't submit these
changes to the repo.
There are PDF files in the distribution which Mercurial does not
know how to merge. There are also unit test data files which are
mostly ASCII but a few that are not. Taking the vendor branch as
authoritative is the best default. So, add the following to
.hg/hgrc in the repo so that these files are taken from the other
branch:
[merge-patterns]
**.pdf = internal:other
curl/tests/data/** = internal:other
2. Modifications
Essential modifications to curl distributions.
2.0 CMake is now used to configure the build.
Many of the options listed below can now be specified from curl/CMakeLists.txt.
These options should be set as follows:
BUILD_CURL_EXE = ON
BUILD_CURL_TESTS = ON
CURL_STATICLIB = ON
ENABLE_ARES = OFF
ENABLE_THREADED_RESOLVER = OFF
Also curl/CMake/FindOpenSSL.cmake and curl/CMake/FindZLIB.cmake are customized
to find only the versions installed in the packages directory.
2.1 Do Not Use C-Ares for DNS Lookups
Use the threaded resolver system which is built on the platform's
resolver library and made asynchronous and thread safe. The
c-ares library appears to be at the heart of many of the DNS
lookup failures customers see.
Changes:
* curl/lib/config-win32.h modified to enable threaded resolver.
* curl/src/config-win32.h modified identically.
* configure script invoked with correct arguments to use the
threaded resolver on linux and darwin.
There is some scripting in the build-cmd.sh file to try to catch
failures in this area but don't rely on it.
2.2 Do Not Modify include/curl/curlbuild.h
The curlbuild.h file distributed in the source tarball is
specially constructed for platforms that cannot run configure and
probe the environment (i.e. Windows). If you update the vendor
branch and then merge back with the default, be certain that
curlbuild.h is copied verbatim from the source distribution and is
not modified in subsequent commits. Running configure on linux or
mac will edit the file, removing the windows specific parts.
2.3 lib/Makefile.vc10 Modified
Additional makefile configurations added for our specific type of
library dependencies: 'release-ssl-dll-zlib' (release build,
static libcurl, dynamic ssl, static zlib) and 'debug-ssl-dll-zlib'
(debug build, static libcurl, dynamic ssl, static zlib).
This file has been edited by lindenlab to pick up the
LINDEN_LIBPATH variable and find our autobuild-packaged library
files. If you upgrade curl, be sure to carry those modifications
forward, or find another way to link in the autobuild libraries.
Debug and Release builds for all targets use /Z7 flag (and lost
any /Gm flag) anticipating static library builds.
2.4 src/Makefile.vc10 Modified
Similar changes to build curl.exe on Windows. Only a
release-ssl-dll-zlib configuration added but it can be used to
build either a 'release' or 'debug' target curl.exe.
2.5 Imperfect Mercurial repository work
The path I took to canonicalize the branch and source code
directory structure, at changesets 1b274a267882 and 4a82c402b11f,
was not the optimal way. What I did was basically:
* hg update vendor
* hg mv curl-version curl
* hg ci
* tar xzf
* hg addremove
* hg ci
* hg update default
* hg mv curl-version curl
* hg ci
* hg merge vendor
* hg ci
What should have happened (and what is happening for other
libraries) involves a single 'hg mv' operation:
* hg update vendor
* hg mv curl-version curl
* hg ci
* hg update default
* hg merge vendor (will do the rename)
* hg ci
* hg update vendor
* tar xzf
* hg addremove
* hg ci
* hg update default
* hg merge vendor
* hg ci
At some point, we might go back and repair/replace the repo.
2.6 Unit Test Changes
Bumped timestamp in tests/data/test172 so test runs. Not
certain how this ever worked, it's stale by several years.
Watch for fixes from upstream and restore the file when
possible. (Fixed in 7.36.0)
2.7 Pipelining Data Corruption
Timed out requests involved in pipelines may result in loss of
response stream synchronization when other requests follow the
timed out requests. This has been reported to the curl devs as
bug 1420 and some defensive measures are taken in the Linden
'default' branch until an official fix is available. Once it is
available, these changes (a8784f6e153f and 46fd1a2376ab) should be
reverted.
I recommend testing any 1420 fix or workaround. The process isn't
entirely deterministic. But some hooks are in place to make this
less painful.
1. Edit llcorehttp/_httpoprequest.cpp to force a request timeout
of one second:
diff -r 0e3365c1843d indra/llcorehttp/_httpoprequest.cpp
--- a/indra/llcorehttp/_httpoprequest.cpp Mon Oct 06 15:22:58 2014 -0400
+++ b/indra/llcorehttp/_httpoprequest.cpp Fri Oct 10 14:31:16 2014 -0400
@@ -626,8 +626,8 @@
// *TODO: Find a better scheme than timeouts to guarantee liveness.
xfer_timeout *= cpolicy.mPipelining;
}
- // *DEBUG: Useful for timeout handling and "[curl:bugs] #1420" tests
- // xfer_timeout = 1L;
+ // *DEBUG: Enable following override for timeout handling and "[curl:bugs] #1420" tests
+ xfer_timeout = 1L;
code = curl_easy_setopt(mCurlHandle, CURLOPT_TIMEOUT, xfer_timeout);
check_curl_easy_code(code, CURLOPT_TIMEOUT);
code = curl_easy_setopt(mCurlHandle, CURLOPT_CONNECTTIMEOUT, timeout);
2. Edit llcorehttp/httpcommon.cpp to disable request retry for
the Core_9 (invalid status) error:
diff -r 0e3365c1843d indra/llcorehttp/httpcommon.cpp
--- a/indra/llcorehttp/httpcommon.cpp Mon Oct 06 15:22:58 2014 -0400
+++ b/indra/llcorehttp/httpcommon.cpp Fri Oct 10 14:31:16 2014 -0400
@@ -254,8 +254,9 @@
*this == op_timedout || // Timer expired
*this == post_error || // Transport problem
*this == partial_file || // Data inconsistency in response
- *this == inv_cont_range || // Short data read disagrees with content-range
- *this == inv_status); // Inv status can reflect internal state problem in libcurl
+ // *DEBUG: Comment out 'inv_status' test for [curl:bugs] #1420 testing.
+ // *this == inv_status || // Inv status can reflect internal state problem in libcurl
+ *this == inv_cont_range); // Short data read disagrees with content-range
}
} // end namespace LLCore
3. Edit llcorehttp/_httppolicy.cpp and add some chatty logging
whenever a request timeout error is generated:
diff -r 0e3365c1843d indra/llcorehttp/_httppolicy.cpp
--- a/indra/llcorehttp/_httppolicy.cpp Mon Oct 06 15:22:58 2014 -0400
+++ b/indra/llcorehttp/_httppolicy.cpp Fri Oct 10 15:10:12 2014 -0400
@@ -414,6 +414,18 @@
// Retry or finalize
if (! op->mStatus)
{
+ // *DEBUG: For "[curl:bugs] #1420" tests. This will interfere
+ // with unit tests due to allocation retention by logging code.
+ // But you won't be checking this in enabled.
+#if 1
+ if (op->mStatus == HttpStatus(HttpStatus::EXT_CURL_EASY, CURLE_OPERATION_TIMEDOUT))
+ {
+ LL_WARNS(LOG_CORE) << "HTTP request " << static_cast<HttpHandle>(op)
+ << " timed out."
+ << LL_ENDL;
+ }
+#endif
+
// If this failed, we might want to retry.
if (op->mPolicyRetries < op->mPolicyRetryLimit && op->mStatus.isRetryable())
{
4. Build and visit mesh- and texture-rich regions. Monitor the
SecondLife.log file for the 'timed out' log message and Core_9
errors. The test isn't deterministic but you want *NO* Core_9
errors and *many* 'timed out' messages:
2014-10-10T19:03:15Z WARNING: LLCore::HttpPolicy::stageAfterCompletion: HTTP request 2CC9FCB0 timed out.
2014-10-10T19:03:15Z WARNING: LLCore::HttpPolicy::stageAfterCompletion: HTTP request 24DA9FC0 timed out.
2014-10-10T19:03:15Z WARNING: LLCore::HttpPolicy::stageAfterCompletion: HTTP request 2CC9FD90 timed out.
2014-10-10T19:03:15Z WARNING: LLCore::HttpPolicy::stageAfterCompletion: HTTP request 26D68718 timed out.
2014-10-10T19:03:15Z WARNING: LLCore::HttpPolicy::stageAfterCompletion: HTTP request 26D67F38 timed out.
2014-10-10T19:03:15Z WARNING: LLCore::HttpPolicy::stageAfterCompletion: HTTP request 26D68C58 timed out.
2014-10-10T19:03:15Z WARNING: LLCore::HttpPolicy::stageAfterCompletion: HTTP request 252218F8 timed out.
5. Follow this test with a Wireshark verification that pipelining
is still working.
3. Source Origin
7.47.0:
https://github.com/bagder/curl.git
(tag:curl-7_47_0)
7.42.1:
https://github.com/bagder/curl.git
(tag:curl-7_42_1)
7.38.0:
http://curl.haxx.se/download/curl-7.38.0.tar.gz
7.37.0:
http://curl.haxx.se/download/curl-7.37.0.tar.gz
7.36.0:
http://curl.haxx.se/download/curl-7.36.0.tar.gz
7.34.0:
http://curl.haxx.se/download/curl-7.34.0.tar.gz
4. Package Contents
Common:
* include/curl/curl.h
* include/curl/curlbuild.h
* include/curl/curlrules.h
* include/curl/curlver.h
* include/curl/easy.h
* include/curl/mprintf.h
* include/curl/multi.h
* include/curl/stdcheaders.h
* include/curl/typecheck-gcc.h
Windows:
* lib/release/libcurl.lib
* lib/debug/libcurld.lib
Mac OS X (debug and release):
* lib/release/libcurl.a
* lib/debug/libcurl.a
Linux:
* lib/release/libcurl.a
* lib/debug/libcurl.a
5. Known Dependencies
This is not authoritative, it's just a reminder to followup
with dependent packages.
* openjpeg (*should* be a dependency when building certain tools)
* viewer
A. Appendix A - Notes on 'vendor' branch maintenance
7.36.0 from 7.34.0.
This wasquite a bit easier than the effort below. With the
following three commands, only one deleted file ended up being
split into two new files. Good enough for this effort.
* hg addremove -X curl/docs/examples/pop3slist.c -X curl/docs/examples/imap.c -s 60
* hg rm curl/docs/examples/imap.c
* hg rm curl/docs/examples/pop3slist.c
7.34.0 from 7.24.0.
Keeping file continuity in the 'vendor' branch was challenging.
The library had been reorganized (for the better) and that
confused 'hg addremove' more than usual. This describes what was
found during the process and hopefully forewarns the next
developer.
After laying down the 7.34.0 source, a simple 'addremove' run
in dry-run mode produces a long list of add and remove operations
*and* a substantial number of suspect renames:
$ hg addremove -s 60 -n
[ long list of adds and removes deleted ]
recording removal of curl/lib/vc6libcurl.dsw as rename to curl/vs/vc6/lib/vc6libcurl.dsw (100% similar)
recording removal of curl/Makefile.msvc.names as rename to curl/winbuild/Makefile.msvc.names (100% similar)
recording removal of curl/lib/curl_rand.h as rename to curl/lib/hostcheck.h (78% similar)
recording removal of curl/lib/curl_rand.h as rename to curl/tests/unit/unit1330.c (79% similar)
recording removal of curl/lib/msvcproj.foot as rename to curl/vs/t/lib/vc6_libcurl_dsp.foot (83% similar)
recording removal of curl/lib/curl_rand.h as rename to curl/tests/server/server_setup.h (79% similar)
recording removal of curl/lib/curl_rand.h as rename to curl/tests/libtest/lib558.c (67% similar)
recording removal of curl/lib/curl_rand.h as rename to curl/lib/http2.h (71% similar)
recording removal of curl/tests/data/test1323 as rename to curl/tests/data/test931 (88% similar)
recording removal of curl/tests/data/test1322 as rename to curl/tests/data/test875 (88% similar)
recording removal of curl/lib/curl_rand.h as rename to curl/lib/bundles.h (63% similar)
recording removal of curl/tests/data/test1324 as rename to curl/tests/data/test829 (88% similar)
recording removal of curl/vc6curl.dsw as rename to curl/vs/vc6/vc6curl.dsw (92% similar)
recording removal of curl/lib/setup_once.h as rename to curl/lib/curl_setup_once.h (92% similar)
recording removal of curl/src/vc6curlsrc.dsw as rename to curl/vs/vc6/src/vc6curltool.dsw (88% similar)
recording removal of curl/lib/curl_rand.h as rename to curl/lib/polarssl_threadlock.h (61% similar)
recording removal of curl/Android.mk as rename to curl/packages/Android/Android.mk (96% similar)
recording removal of curl/lib/krb4.h as rename to curl/lib/curl_sec.h (86% similar)
recording removal of curl/src/hugehelp.h as rename to curl/lib/dotdot.h (83% similar)
recording removal of curl/tests/data/test1324 as rename to curl/tests/data/test1409 (60% similar)
recording removal of curl/lib/curl_rand.h as rename to curl/tests/libtest/lib1508.c (75% similar)
recording removal of curl/lib/setup.h as rename to curl/lib/curl_setup.h (89% similar)
recording removal of curl/src/hugehelp.h as rename to curl/src/tool_hugehelp.h (83% similar)
recording removal of curl/docs/examples/htmltitle.cc as rename to curl/docs/examples/htmltitle.cpp (98% similar)
Of these, lib/curl_rand.*, src/hugehelp.h, lib/krb4.h,
tests/data/test815 should not be renamed (they're actually being
deleted). Update the dry-run command with some exclusions:
$ hg addremove -X curl/lib/curl_rand.h -X curl/lib/curl_rand.c \
-X curl/src/hugehelp.h -X curl/lib/krb4.h \
-X curl/tests/data/test815 -s 60 -n
recording removal of curl/lib/vc6libcurl.dsw as rename to curl/vs/vc6/lib/vc6libcurl.dsw (100% similar)
recording removal of curl/Makefile.msvc.names as rename to curl/winbuild/Makefile.msvc.names (100% similar)
recording removal of curl/src/vc6curlsrc.dsw as rename to curl/vs/vc6/src/vc6curltool.dsw (88% similar)
recording removal of curl/src/tool_cb_skt.h as rename to curl/lib/hostcheck.h (73% similar)
recording removal of curl/lib/setup.h as rename to curl/lib/curl_setup.h (89% similar)
recording removal of curl/tests/data/test1322 as rename to curl/tests/data/test875 (88% similar)
recording removal of curl/src/tool_cb_skt.h as rename to curl/tests/unit/unit1330.c (74% similar)
recording removal of curl/src/tool_cb_skt.h as rename to curl/src/tool_hugehelp.h (76% similar)
recording removal of curl/lib/setup_once.h as rename to curl/lib/curl_setup_once.h (92% similar)
recording removal of curl/lib/msvcproj.foot as rename to curl/vs/t/lib/vc6_libcurl_dsp.foot (83% similar)
recording removal of curl/src/tool_cb_skt.h as rename to curl/lib/bundles.h (60% similar)
recording removal of curl/tests/data/test1324 as rename to curl/tests/data/test829 (88% similar)
recording removal of curl/src/tool_cb_skt.h as rename to curl/tests/server/server_setup.h (74% similar)
recording removal of curl/vc6curl.dsw as rename to curl/vs/vc6/vc6curl.dsw (92% similar)
recording removal of curl/src/tool_cb_skt.h as rename to curl/tests/libtest/lib558.c (64% similar)
recording removal of curl/src/tool_cb_skt.h as rename to curl/lib/http2.h (67% similar)
recording removal of curl/src/tool_cb_skt.h as rename to curl/lib/dotdot.h (77% similar)
recording removal of curl/tests/data/test1323 as rename to curl/tests/data/test931 (88% similar)
recording removal of curl/docs/examples/htmltitle.cc as rename to curl/docs/examples/htmltitle.cpp (98% similar)
recording removal of curl/tests/data/test1324 as rename to curl/tests/data/test1409 (60% similar)
recording removal of curl/src/tool_cb_skt.h as rename to curl/tests/libtest/lib1508.c (70% similar)
recording removal of curl/Android.mk as rename to curl/packages/Android/Android.mk (96% similar)
Better. Now add src/tool_cb_skt.h and lib/msvcproj.foot (which
are also being deleted):
$ hg addremove -X curl/lib/curl_rand.h -X curl/lib/curl_rand.c \
-X curl/src/hugehelp.h -X curl/lib/krb4.h \
-X curl/tests/data/test815 -X curl/src/tool_cb_skt.h \
-X curl/lib/msvcproj.foot -s 60 -n
recording removal of curl/lib/vc6libcurl.dsw as rename to curl/vs/vc6/lib/vc6libcurl.dsw (100% similar)
recording removal of curl/Makefile.msvc.names as rename to curl/winbuild/Makefile.msvc.names (100% similar)
recording removal of curl/src/vc6curlsrc.dsw as rename to curl/vs/vc6/src/vc6curltool.dsw (88% similar)
recording removal of curl/docs/examples/printf_macro.h as rename to curl/lib/hostcheck.h (66% similar)
recording removal of curl/lib/setup.h as rename to curl/lib/curl_setup.h (89% similar)
recording removal of curl/tests/data/test1322 as rename to curl/tests/data/test875 (88% similar)
recording removal of curl/docs/examples/printf_macro.h as rename to curl/tests/unit/unit1330.c (67% similar)
recording removal of curl/docs/examples/printf_macro.h as rename to curl/src/tool_hugehelp.h (68% similar)
recording removal of curl/docs/examples/printf_macro.h as rename to curl/tests/libtest/lib1508.c (64% similar)
recording removal of curl/Android.mk as rename to curl/packages/Android/Android.mk (96% similar)
recording removal of curl/tests/data/test1324 as rename to curl/tests/data/test829 (88% similar)
recording removal of curl/docs/examples/printf_macro.h as rename to curl/tests/server/server_setup.h (67% similar)
recording removal of curl/vc6curl.dsw as rename to curl/vs/vc6/vc6curl.dsw (92% similar)
recording removal of curl/docs/examples/printf_macro.h as rename to curl/lib/dotdot.h (70% similar)
recording removal of curl/docs/examples/printf_macro.h as rename to curl/lib/http2.h (61% similar)
recording removal of curl/tests/data/test1323 as rename to curl/tests/data/test931 (88% similar)
recording removal of curl/docs/examples/htmltitle.cc as rename to curl/docs/examples/htmltitle.cpp (98% similar)
recording removal of curl/tests/data/test1324 as rename to curl/tests/data/test1409 (60% similar)
recording removal of curl/lib/setup_once.h as rename to curl/lib/curl_setup_once.h (92% similar)
Add docs/examples/printf_macro.h (also deleting):
$ hg addremove -X curl/lib/curl_rand.h -X curl/lib/curl_rand.c \
-X curl/src/hugehelp.h -X curl/lib/krb4.h \
-X curl/tests/data/test815 -X curl/src/tool_cb_skt.h \
-X curl/lib/msvcproj.foot -X curl/docs/examples/printf_macro.h \
-s 60 -n
recording removal of curl/lib/vc6libcurl.dsw as rename to curl/vs/vc6/lib/vc6libcurl.dsw (100% similar)
recording removal of curl/Makefile.msvc.names as rename to curl/winbuild/Makefile.msvc.names (100% similar)
recording removal of curl/src/vc6curlsrc.dsw as rename to curl/vs/vc6/src/vc6curltool.dsw (88% similar)
recording removal of curl/lib/setup.h as rename to curl/lib/curl_setup.h (89% similar)
recording removal of curl/tests/data/test1322 as rename to curl/tests/data/test875 (88% similar)
recording removal of curl/tests/data/test1324 as rename to curl/tests/data/test829 (88% similar)
recording removal of curl/vc6curl.dsw as rename to curl/vs/vc6/vc6curl.dsw (92% similar)
recording removal of curl/Android.mk as rename to curl/packages/Android/Android.mk (96% similar)
recording removal of curl/tests/data/test1323 as rename to curl/tests/data/test931 (88% similar)
recording removal of curl/docs/examples/htmltitle.cc as rename to curl/docs/examples/htmltitle.cpp (98% similar)
recording removal of curl/tests/data/test1324 as rename to curl/tests/data/test1409 (60% similar)
recording removal of curl/lib/setup_once.h as rename to curl/lib/curl_setup_once.h (92% similar)
Exclude the test files (being deleted):
$ hg addremove -X curl/lib/curl_rand.h -X curl/lib/curl_rand.c \
-X curl/src/hugehelp.h -X curl/lib/krb4.h \
-X curl/tests/data/test815 -X curl/src/tool_cb_skt.h \
-X curl/lib/msvcproj.foot -X curl/docs/examples/printf_macro.h \
-X curl/tests/data/test875 -X curl/tests/data/test829 \
-X curl/tests/data/test931 -X curl/tests/data/test1409 -s 60 -n
recording removal of curl/lib/vc6libcurl.dsw as rename to curl/vs/vc6/lib/vc6libcurl.dsw (100% similar)
recording removal of curl/Makefile.msvc.names as rename to curl/winbuild/Makefile.msvc.names (100% similar)
recording removal of curl/src/vc6curlsrc.dsw as rename to curl/vs/vc6/src/vc6curltool.dsw (88% similar)
recording removal of curl/lib/setup.h as rename to curl/lib/curl_setup.h (89% similar)
recording removal of curl/vc6curl.dsw as rename to curl/vs/vc6/vc6curl.dsw (92% similar)
recording removal of curl/Android.mk as rename to curl/packages/Android/Android.mk (96% similar)
recording removal of curl/docs/examples/htmltitle.cc as rename to curl/docs/examples/htmltitle.cpp (98% similar)
recording removal of curl/lib/setup_once.h as rename to curl/lib/curl_setup_once.h (92% similar)
That looks like a correct list of renames. Execute it for effect
then check status:
$ hg addremove -X curl/lib/curl_rand.h -X curl/lib/curl_rand.c \
-X curl/src/hugehelp.h -X curl/lib/krb4.h \
-X curl/tests/data/test815 -X curl/src/tool_cb_skt.h \
-X curl/lib/msvcproj.foot -X curl/docs/examples/printf_macro.h \
-X curl/tests/data/test875 -X curl/tests/data/test829 \
-X curl/tests/data/test931 -X curl/tests/data/test1409 -s 60
recording removal of curl/lib/vc6libcurl.dsw as rename to curl/vs/vc6/lib/vc6libcurl.dsw (100% similar)
recording removal of curl/Makefile.msvc.names as rename to curl/winbuild/Makefile.msvc.names (100% similar)
recording removal of curl/src/vc6curlsrc.dsw as rename to curl/vs/vc6/src/vc6curltool.dsw (88% similar)
recording removal of curl/lib/setup.h as rename to curl/lib/curl_setup.h (89% similar)
recording removal of curl/vc6curl.dsw as rename to curl/vs/vc6/vc6curl.dsw (92% similar)
recording removal of curl/Android.mk as rename to curl/packages/Android/Android.mk (96% similar)
recording removal of curl/docs/examples/htmltitle.cc as rename to curl/docs/examples/htmltitle.cpp (98% similar)
recording removal of curl/lib/setup_once.h as rename to curl/lib/curl_setup_once.h (92% similar)
$ hg status
! curl/docs/examples/printf_macro.h
! curl/lib/curl_rand.c
! curl/lib/curl_rand.h
! curl/lib/krb4.h
! curl/lib/msvcproj.foot
! curl/src/hugehelp.h
! curl/src/tool_cb_skt.h
? curl/tests/data/test1409
? curl/tests/data/test829
? curl/tests/data/test875
? curl/tests/data/test931
Those are removed ('!') and added ('?') files that weren't
correctly moved and which now need to 'rm'd and 'add'd,
respectively:
$ hg rm curl/docs/examples/printf_macro.h curl/lib/curl_rand.c \
curl/lib/curl_rand.h curl/lib/krb4.h curl/lib/msvcproj.foot \
curl/src/hugehelp.h curl/src/tool_cb_skt.h
$ hg add curl/tests/data/test{829,875,931,1409}
And now 'hg status' reports no unaccounted for files in the
'vendor' branch.
Next, we switch over to 'default' branch and prepare for the merge.
First, need to disable merging of PDF and test data files as
mentioned above. Add the following to the repo's '.hg/hgrc' file:
[merge-patterns]
**.pdf = internal:other
curl/tests/data/** = internal:other
And now the merge. The full file list is long, I've extracted the
unusual events in the merge here:
$ hg merge vendor
merging curl/Android.mk and curl/packages/Android/Android.mk to curl/packages/Android/Android.mk
merging curl/docs/examples/htmltitle.cc and curl/docs/examples/htmltitle.cpp to curl/docs/examples/htmltitle.cpp
merging curl/lib/Makefile.vc10
warning: conflicts during merge.
merging curl/lib/setup.h and curl/lib/curl_setup.h to curl/lib/curl_setup.h
merging curl/lib/setup_once.h and curl/lib/curl_setup_once.h to curl/lib/curl_setup_once.h
merging curl/src/vc6curlsrc.dsw and curl/vs/vc6/src/vc6curltool.dsw to curl/vs/vc6/src/vc6curltool.dsw
merging curl/vc6curl.dsw and curl/vs/vc6/vc6curl.dsw to curl/vs/vc6/vc6curl.dsw
Check each of the three-way merges and verify that they make
sense. That the underlying file movement by Hg was valid and that
the resultant file make sense. In these cases, all merges appear
valid.
The conflicted .vc10 file requires real attention. But that's
mostly manual editing to get it back into shape with
Linden-specific actions and options. Roughly, the following were
done in the merge:
* Keep '/DUSE_SCHANNEL' from CFLAGSWINSSL. Selects native
ssl/tls on Windows which we're not using (yet).
* Add '/DUSE_OPENSSL' from new 7.34.0 drop to CFLAGSSSL.
* CFLAGSZLIB changed to $(LINDEN_INCPATH)
* ZLIBLIB to $(LINDEN_LIBPATH)/$(ZLIB_NAME)
* Didn't keep the 'advapi32.lib' in WINLIBS which came from
the 7.34.0 distribution. May be needed later, seems to link
and run curl.exe fine without.
* LFLAGSZLIB in all/most configurations to
'/LIBPATH:$(LINDEN_LIBPATH)'.
* LFLAGSSSL to '/LIBPATH:$(OPENSSL_PATH)\out32dll' in the
'release-ssl-zlib' configuration which I think was pointless.
* Drop RESOURCE from 'release-ssl-ssh2-zlib' and
'debug-ssl-ssh2-zlib'.
* TARGET modified on some configurations. Think this was
unnecessary.
Finally, the 'hg addremove' process seemed to get a little lost in
merging file removals on the vendor branch over to the default branch.
In the end, I did a plain diff of the default branch 'curl' directory
against a clean unpack of the curl distribution and found I had some
deleted files left. I 'hg rm'd these manually:
$ hg rm ....
R curl/docs/examples/printf_macro.h
R curl/lib/config-vms.h
R curl/lib/curl_rand.c
R curl/lib/curl_rand.h
R curl/lib/krb4.c
R curl/lib/krb4.h
R curl/lib/libcurl.imp
R curl/lib/libcurl.vcproj
R curl/lib/msvcproj.foot
R curl/lib/msvcproj.head
R curl/lib/vc6libcurl.dsp
R curl/m4/curl-system.m4
R curl/packages/vms/hpssl_alpha.opt
R curl/packages/vms/hpssl_ia64.opt
R curl/packages/vms/hpssl_vax.opt
R curl/packages/vms/ldap.opt
R curl/packages/vms/openssl_alpha.opt
R curl/packages/vms/openssl_ia64.opt
R curl/packages/vms/openssl_ssl_alpha.opt
R curl/packages/vms/openssl_ssl_ia64.opt
R curl/packages/vms/openssl_ssl_vax.opt
R curl/packages/vms/openssl_vax.opt
R curl/src/config-amigaos.h
R curl/src/config-mac.h
R curl/src/config-riscos.h
R curl/src/config-win32.h
R curl/src/curl_config.h.in
R curl/src/hugehelp.c
R curl/src/hugehelp.h
R curl/src/setup.h
R curl/src/tool_cb_skt.c
R curl/src/tool_cb_skt.h
R curl/src/vc6curlsrc.dsp
R curl/tests/data/test1322
R curl/tests/data/test1323
R curl/tests/data/test1324
And there it is. 7.24.0 upgraded to 7.34.0 with file history
maintained (to an approximation).
===================================================================
Third-Party Library Repo Structure
Introduction
We want to have a way to capture local modifications to a third-party
open-source project, such as libcurl, without needing write access to
their public repository. We want to be able to carry forward such
modifications to newer versions of the public project. All this
should be independent of the organizational decision as to whether
it's even desirable to try to submit our local modifications upstream.
Fortunately, the Subversion folks articulated a process years ago that
addresses this very requirement. They call it "Vendor Branches." The
same tactic, suitably adapted, works with Mercurial too.
The essence of the idea is that we capture and tag a particular
snapshot of the open-source project. We develop our local
modifications to that, and the repository tip incorporates them. But
when we want to update to a newer version of the public project, we
bring it into the repository in such a way that we can discover the
changes from the original snapshot and the new one -- and then have
Mercurial apply those deltas to the ''combined'' source.
The following material is adapted from
http://svnbook.red-bean.com/en/1.1/ch07s05.html, the Red Bean
Subversion book, but recast for Mercurial. The Linden source for this
material is an internal wiki. There may be superceding documentation
on the public wiki when you read this. We recommend searching there
for updates to conventions below. And note that each particular
library may implement variations of this scheme.
General Vendor Branch Management Procedure
Managing vendor branches generally works like this. You create a
named branch ("vendor") to store the vendor source snapshots. Then
you import the third party code into that branch. Your modified
branch (named "default") is based on "vendor". You always make your
local changes to the default branch. With each new release of the
code you are tracking you bring it into the "vendor" branch and merge
the changes into "default", resolving whatever conflicts occur between
your local changes and the upstream changes.
Perhaps an example will help to clarify this algorithm. We'll use a
scenario where your development team is creating a calculator program
that links against a third-party complex number arithmetic library,
libcomplex. We'll construct a repository specifically for our
locally-modified version of that library. To begin, we must
initialize our repository and create at least one file in our
"default" branch.
$ hg init ourcomplex
$ cd ourcomplex
$ touch README.txt
$ hg commit README.txt
Now we can create the vendor branch and do the import of the first
vendor drop. We'll call our vendor branch "vendor", and each
successive code drop will be tagged "current".
$ hg branch vendor
$ tar -xjf ../libcomplex-1.0.tar.bz2
$ mv libcomplex-1.0 libcomplex
$ hg addremove
$ hg commit -m "1.0 source drop"
$ hg tag -r tip current
$ hg tag -r current 1.0
We now have the current version of the libcomplex source code in
branch "vendor", tagged "current" and in a non-version-specific source
code subdirectory ("libcomplex"). Next, we merge it into the default
branch. It is in the default branch that we will make our
customizations.
$ hg update default
$ hg merge vendor
$ hg commit -m "initial: 1.0"
We get to work customizing the libcomplex code. Before we know it,
our modified version of libcomplex is now completely integrated into
our calculator program.
A few weeks later, the developers of libcomplex release a new version
of their library, version 1.1, which contains some features and
functionality that we really want. We'd like to upgrade to this new
version, but without losing the customizations we made to the existing
version. What we essentially would like to do is to replace our
current baseline version of libcomplex 1.0 with a copy of libcomplex
1.1, and then have Mercurial re-apply the custom modifications we
previously made to that library to the new version. But we actually
approach the problem from the other direction, applying the changes
made to libcomplex between versions 1.0 and 1.1 to our modified copy
of it.
To perform this upgrade, we update our repository to our vendor
branch, and update the "current" tag with the new libcomplex 1.1
source code. We quite literally replace the existing files with the
new files, clearing out the whole tree and exploding the libcomplex
1.1 release tarball in its place. The goal here is to make the tip of
our vendor branch contain only the libcomplex 1.1 code, and to ensure
that all that code is under version control. Oh, and we want to do
this with as little version control history disturbance as possible.
$ hg update vendor
$ rm -rf *
$ tar -xjf ../libcomplex-1.1.tar.bz2
$ mv libcomplex-1.1 libcomplex
$ hg addremove -s 60
$ # Additional 'hg add' and 'hg rm' commands if needed
$ hg commit -m "1.1 source drop"
After unpacking the 1.1 tarball, hg status will show files with local
modifications as well as, perhaps, some unversioned or missing files.
If we did what we were supposed to do, the unversioned files are only
those new files introduced in the 1.1 release of libcomplex. The
missing files are files that were in 1.0 but not in 1.1. The 'hg
addremove' command deals with both, and more: the '-s 60' switch
directs Mercurial to compare added files to deleted files, recognizing
any file at least 60% similar as a move/rename.
For simple or stable libraries, the 'hg addremove' command should be
reliable. For more complicated libraries subject to refactoring or
large gaps of time between updates (e.g. libcurl), it can get a little
lost trying to match files in the old release with files in the new
release. Pay attention to the output of the command or better still,
do dry runs. Files erroneously moved can be excluded with the '-X'
option and then dealt with individually with 'hg add' and 'hg rm'
commands after 'hg addremove'. (The readme file in the curl library
should document a particularly challenging case.)
The 'addremove' process doesn't have to be perfect. Recreating the
evolution of the upstream source tree isn't universally practical.
But we'd like to capture movement of files in the vendor branch that
are modified in the default branch. If achieving that becomes too
tedious, then re-implementation of the default branch edit in a new
file is fine. Just note it here for the next developer.
Finally, once our current working copy contains only the libcomplex
1.1 code, we commit the changes we made to get it looking that way.
Our current vendor branch now contains the new vendor drop. We move
the 'current' tag to the new version (in the same way we previously
tagged the version 1.0 vendor drop), and then merge the differences
between the version 1.0 and version 1.1 into our default branch.
$ hg tag -f -r tip current
$ hg tag -r current 1.1
$ hg update default
$ hg merge vendor
# resolve all the conflicts between their changes and our changes
# if you will have conflicts in .hgtags, simply take *all* lines
...
$ hg commit -m "update with 1.1"
Any additional work needed to get the merged library working can
now be done on the default branch.
Revision Tags
We don't currently make use of Mercurial tags in the build and release
process for 3rd-party libraries. But we would like to establish a
convention to document update and release points. The tags we would
like to establish are:
* 'current' Points to a succession of vendor releases checked into
the 'vendor' branch. Will almost always be at or close to branch
head.
* '<version>' Tag on the 'vendor' branch pointing to a verbatim
checkin of a 3rd-party's <version> release. Example: '7.21.1' for
a particular version of libcurl we have used.
* Release-type tags on the default branch aren't as useful given how
Mercurial handles tags and how autobuild works.
Schematic of a Third-Party Repository
Below is the output of the 'hg glog' command showing a library project
going through an initial 1.0 release and an update from the vendor to
1.1. Significant revisions in the repository lifecycle are as
follows:
0 Creation of the repo with an initial file.
1 1.0 code drop on branch 'vendor'
4 Merge of 1.0 code onto branch 'default'
5 Modifications to library we wish to keep over time. Released.
6 1.1 code drop on branch 'vendor'
9 Merge of 1.1 code onto branch 'default'
10 Fixes to merge yielding production 1.1 library. Released.
@ changeset: 10:888229641f6e
| tag: tip
| user: Monty Brandenberg <[email protected]>
| date: Wed Oct 30 13:35:51 2013 -0400
| summary: Work to get 1.1 merge working. Release.
|
o changeset: 9:925ccdf09f50
|\ parent: 5:83c5775c23dc
| | parent: 8:977001a08e48
| | user: Monty Brandenberg <[email protected]>
| | date: Wed Oct 30 13:35:20 2013 -0400
| | summary: update with 1.1
| |
| o changeset: 8:977001a08e48
| | branch: vendor
| | user: Monty Brandenberg <[email protected]>
| | date: Wed Oct 30 13:33:49 2013 -0400
| | summary: Added tag 1.1 for changeset 5f6cb89add91
| |
| o changeset: 7:59bce0f6d12f
| | branch: vendor
| | user: Monty Brandenberg <[email protected]>
| | date: Wed Oct 30 13:33:41 2013 -0400
| | summary: Added tag current for changeset 5f6cb89add91
| |
| o changeset: 6:5f6cb89add91
| | branch: vendor
| | tag: current
| | tag: 1.1
| | parent: 3:8525ad934ecd
| | user: Monty Brandenberg <[email protected]>
| | date: Wed Oct 30 13:33:29 2013 -0400
| | summary: 1.1 source drop
| |
o | changeset: 5:83c5775c23dc
| | tag: 1.0
| | user: Monty Brandenberg <[email protected]>
| | date: Wed Oct 30 13:32:31 2013 -0400
| | summary: Linden-specific changes to the library. Release
| |
o | changeset: 4:bccb736585f4
|\| parent: 0:400e4516c406
| | parent: 3:8525ad934ecd
| | user: Monty Brandenberg <[email protected]>
| | date: Wed Oct 30 13:31:40 2013 -0400
| | summary: initial: 1.0
| |
| o changeset: 3:8525ad934ecd
| | branch: vendor
| | user: Monty Brandenberg <[email protected]>
| | date: Wed Oct 30 13:30:21 2013 -0400
| | summary: Added tag 1.0 for changeset 8ac3828d03bb
| |
| o changeset: 2:7aa1a1cb62d9
| | branch: vendor
| | user: Monty Brandenberg <[email protected]>
| | date: Wed Oct 30 13:30:14 2013 -0400
| | summary: Added tag current for changeset 8ac3828d03bb
| |
| o changeset: 1:8ac3828d03bb
|/ branch: vendor
| tag: 1.0
| user: Monty Brandenberg <[email protected]>
| date: Wed Oct 30 13:30:09 2013 -0400
| summary: 1.0 source drop
|
o changeset: 0:400e4516c406
user: Monty Brandenberg <[email protected]>
date: Wed Oct 30 13:29:16 2013 -0400
summary: Created repo with initial readme file