# Re-execution replication ## 1. What I ran I wrote my own driver `replication/replicate.mjs` and my own tool `replication/parse-head.mjs` (according to section 2.5), and I built the calibration fixtures with the script `replication/make-fixtures.py`. I ran the driver four times; each run had an internal limit of seven minutes and stopped by itself when it reached it. Every row is written to `replication/out/rows.jsonl` at the moment it comes into existence, and the next run skips what is already in the file. I had to kill the first run after 898 rows, the second ran out on its own limit, the third completed the rest. **1884 rows, 1884 expected units, zero missing, zero duplicates, zero surplus.** I did the check against `replication/inputs/files.jsonl`: 300 files of frame W and 300 files of frame S give exactly 1884 units with the arms from section 2.1 and section 2.7. After the run I fixed a bug in my own tool in two cases and **re-measured 286 rows** (the archive of the original rows is in `replication/out/superseded-rows-instrument-v1.jsonl`; the name is English because it is a file name). These were 268 rows of the `tail` arm and 18 webp rows of frame S. The numbers below come from the corrected version. The distribution of rows by state: 1702 measured, 155 with `parse_error` (138 PNG, 8 webp, 5 TIFF, 4 JPEG), 22 excluded for content type, 5 failed fetches. The first version of the tool returned 249 rows in the `tail` arm with `parse_error: tail_unidentifiable` and, for 18 webp rows of frame S, `parse_error: bad_byte_order`; I re-measured all 286 after the fix, and in the corrected version these two reasons do not occur even once. ## 2. The main numbers The share is the share of files where `gps_coord_tag_present AND NOT gps_coord_all_zero` holds (section 2.10). | arm | share | 95 % interval | denominator | |---|---|---|---| | W `original` | 12.5 % | 9.1 – 16.9 % | 273 | | W `display` | 0 % | 0 – 1.3 % | 286 | | W `display_w200` | 0 % | 0 – 11.4 % | 30 | | W `display_w1200` | 0 % | 0 – 11.7 % | 29 | | W `archive_oldest` | 0 % | 0 – 79.3 % | 1 | | W `tail` | 0 % | 0 – 1.5 % | 260 | | S `s_full` | 0 % | 0 – 1.5 % | 252 | | S `s_thumb` | 0 % | 0 – 1.5 % | 259 | | S `s_remote` | 0 % | 0 – 1.5 % | 245 | The paired difference for frame W, across all 242 eligible pairs: `original` 14.0 %, `display` 0 %, a difference of **+14.0 percentage points** (95 % interval +9.7 to +18.4), 34 undecided pairs. By container: 234 pairs within a single container and +14.5 points, 8 pairs across containers and zero. The restriction to pairs smaller than 131 072 bytes has only 6 pairs and a difference of zero. Validation: `tail` compared 262 pairs and disagrees in 75 cases, that is **28.6 %**; the union rule from section 2.7 did not add a single case. `full` for frame W compared 29 pairs and disagrees once (3.4 %), for frame S it compared 30 pairs and disagrees six times (20.0 %). The share from the header is therefore published as a **lower bound**, per section 2.7. The other fields for the measured originals of frame W (273 rows): `gps_ifd_present` is true 59 times, `gps_coord_tag_present` 38 times, of which 34 times with non-zero coordinates, `gps_coord_all_zero` 149 times out of the full 300 rows, `xmp_present` 110 times, `serial_tag_present` 35 times. `icc_present` is true 96 times, and of those 96 the profile description is from the list of common ones (`icc_desc_is_stock`) in 85 cases. `make_present` is true 159 times, and of those 159 the make belongs to consumer devices in 143 cases, so `make_is_consumer_device` has a denominator of 159, not 273. ## 3. Where the method does not match what it describes This is the main output. I mark the steps by section number. **Section 2.10, the denominator.** The method fixes the numerator and says nothing about the denominator. Whether it is counted from the measured files or from all 300 accepted ones is written nowhere, and the difference is 12.5 % against 11.3 %. I chose the measured files. **Section 2.1, the `archive_oldest` arm.** The table says „pre-registered 30-file subsample“, but the list of files carries `archived_url` for only 13 files, and of those only a single one is in the subsample (`accepted_order` 10). The arm is therefore measured once, not thirty times, and one of the nine arms in the table above has a denominator of 1. Similarly, the table calls the URL `imageinfo.archived[last].url`, while the input field is called `archived_url`; the method nowhere defines whether `last` means the oldest or the newest version. **Sections 2.1 and 2.4, `display_is_original` and the width arms.** For 30 of the 300 files of frame W, the URL for width 800 has the same path as the original and differs only in a tracking parameter, that is, exactly the state that `[A-1.3]` describes. In the thirty-file subsample there are six such files. At width 200 there is a single one, and at width 1200 there are nine, which means these two arms measure genuinely resampled images and are not a mostly guaranteed tie. The method meanwhile formulates the rule „`display_is_original` is excluded from every paired statistic“ for the `original`–`display` pair and leaves it open whether the same exclusion should also be applied to the width arms; I applied it to them, because `[A-1.3]` speaks of „every pair“. **Sections 2.7 and 2.8, the `full` rows.** The method says „Every unit is fetched once“ and then prescribes a ten-percent subsample with a complete download. For frame S, those ten percent are on top of what the `s_full` arm already downloads in full anyway, so the 30 `S|s_full|full` rows are redundant. The method also does not say whether the full fetch is truncated to 131 072 bytes. I should not have truncated it, otherwise the cross-check would compare two identical truncated buffers, which is exactly what `[A-2.8]` criticizes about the first revision. **Section 2.5, `parse_error` for large files.** The rule „a segment length that would run past the buffer“ is a `parse_error`. An ordinary PNG with an image larger than the window has a chunk that overflowed, so 138 rows are „undetermined“ only because the image is large, not because something is broken. The method does not distinguish „the chunk overflowed, the metadata are before it and have been read“ from „the metadata may be beyond the window“. I chose the stricter reading, that is, `parse_error` for both cases, and that is why 155 rows are outside the denominator. **Section 2.5, `read_complete`.** The method lists this field in section 2.12 but nowhere defines it. I compute it from `Content-Range` (`bytes 0-131071/2507196` means an incomplete read) and I take a missing header as undetermined. For frame S this is 388 rows with `false`, which is the majority of the arm; neither rule 2.9 nor 2.11 does anything with that field, so it is a dead value. **Section 2.5, `tail`.** The method nowhere says how the window at the end of the file is parsed. That window starts in the middle of a chunk, so no container traversal is possible; I wrote my own scanning of regions. The second thing: the method lists three fields that require reading values, but names four (`gps_coord_all_zero`, `gps_tag_count`, `make_is_consumer_device`, `icc_desc_is_stock`). **Section 2.5, missing tags.** For a missing `Make` and for a missing ICC profile description, the method says that a failure must not look like an absence, but it does not say whether that is `false` or `null`. I chose `null`, and in the table above this shows up as `make_is_consumer_device` having, across the 273 measured originals, a denominator of 159, that is, only those where the make really is present. **Section 2.5, GIF and SVG.** According to the method, the tool knows JPEG, PNG, WebP and TIFF. Among the inputs there is `image/svg+xml` and `image/gif`, and the method does not say what the container for SVG is. The SVG rows drop out through the content-type rule in section 2.9, so the result is the same, but the reason is different from what the method assumes. The second thing is the `s_thumb` arm: for 299 of the 300 attachments of frame S, the address differs from `s_full` only in the path `original` against `small`, that is, it is a resampled image from the same server. The method nowhere comments on this symmetry, even though for frame W it handles it with the `[A-1.3]` rule. **Section 2.5, `range_honoured`.** The method says „records whether the server answered 206 or 200“ and does not define which value is `true`. I chose `true` for 206, and in dataset.json it is that way. **Section 2.5, `unit_key`.** The scheme `||` is prescribed, but the vocabulary for `kind` is nowhere defined. I chose `head` and `full`; the `tail` arm has `kind=head` and `arm=tail`. A row-by-row comparison with the second session depends on this choice. **Section 2.6, calibration.** The text says „eight fixtures“ and then lists ten of them: four formats in two versions each is eight, plus one JPEG and one PNG beyond the window are two more. Furthermore: the task said that the fixtures would be built by `/usr/bin/python3` with PIL, but PIL 9.5 silently discards a GPS sub-IFD written through `Image.Exif.get_ifd()`; `exif.tobytes()` returns a block without the `0x8825` tag. I therefore had to assemble the fixtures with GPS by hand, byte by byte, which is work that the method attributes to PIL. **Section 2.6, `parse_error` for displaced fixtures.** Whether a displaced JPEG returns a `parse_error` depends on exactly where the cut at 131 072 bytes falls. I built the same construction twice, with GPS at the same position, and they differ only in the alignment of the cut: one returns `parse_error`, the other `false` (the file `_jpeg_displaced_boundary.jpg` in `replication/out/fixtures/`). The calibration therefore tests the alignment of the boundary, not the capability of the tool. When I added the window at the end of the file to the tool, both displaced fixtures were correctly found from the window at the end, that is, the metadata were not lost, they were only beyond the window. **Section 2.7, `tail_disagrees`.** The method counts every disagreement, but the union rule handles only one direction (`tail` finds what `head` missed). On a literal reading the rate is 28.6 %, which is substantially more than five percent, and the share is published as a lower bound. Of the 75 disagreements across 262 comparisons, however, 59 are ones where the header has an IFD and the tail does not, and the remaining 16 are ones where the header could not be parsed and the tail found nothing; the direction „the tail found what the head missed“ is zero. The number 28.6 % therefore mainly measures the fact that the metadata are at the start of the file, which the method itself assumes. In `dataset.json` I state both rates (`rate` and `union_rate`), and in the main numbers I use the literal reading. **Section 2.9, duplicates.** „Duplicate `sha256_head` within one arm“ does not say whether a row is discarded or a file. I kept the row in `rows.jsonl`, excluded it from the share and count it separately (two rows each in the `tail`, `s_full`, `s_thumb`, `s_remote` arms). **Section 2.9, content type.** The rule allows four types. One server typed a JPEG as `image/jpg` and one did not send `Content-Type` at all; both rows are excluded, even though they are images. The method does not anticipate this reason. **Section 2.11, the interval for frame S.** The method asks for an interval over the number of distinct accounts, but the accepted list of files does not contain an account field. The number of accounts cannot be established from this input, and in `dataset.json` the interval is over files, with a note. **Section 2.11, the pairing condition.** „both arms returned 2xx“ excludes nothing in this run, because every successful fetch returned 200 or 206; the real exclusions are `parse_error` and `display_is_original`. It also holds that the `[A-2.2]` restriction to files smaller than 131 072 bytes has, at a share of 12.5 %, only 6 pairs, so the rule about twenty pairs can never trigger and the ordering is decided by the full file. **Section 2.4, failures without a status.** Five `s_remote` rows failed before a response (three unknown hosts, two timeouts). The method knows about retrying on 404, 403, 410 and a timeout; for a network error without a status it says nothing. I retried the timeouts once after thirty seconds; network errors without a status I did not retry, because the method has no rule for them. **Sections 2.2 and 2.3, the sample.** The procedure for drawing random files and ten pages of the timeline is written down in the method, but a re-execution cannot carry it out: the input is a finished list. The random selection has no seed, so the draw cannot be repeated, only taken over. The element `commons_generator` (`MediaWiki 1.47.0-wmf.19`) and the version of the instance (`4.7.0`) are the only verifiable elements, and they reproduced. **Section 2.12, the salt.** The method says that the salt never leaves `data/`. My `out/` is not `data/`, but I wrote the salt into `replication/out/salt.txt`, that is, next to the rows. If this directory is published together with the rows, the hash of the frame S addresses can be broken by brute force, because the addresses have a narrow space. The salt has to be removed before publication; in `rows.jsonl` there is only `url_ref` without the address. ## 4. The denominator for each share - **W `original`.** I count 273 rows, that is, all 300 except 11 excluded for the type `image/svg+xml` and 16 with `parse_error`; excluded and undetermined rows do not go on the „no“ side. - **W `display`.** I count 286 rows, that is, all 300 except 14 with `parse_error`; none was excluded. - **W `display_w200`.** I count all 30 rows of the subsample; none was excluded or undetermined. - **W `display_w1200`.** I count 29 out of 30, one row has a `parse_error`. - **W `archive_oldest`.** I count the single row that exists for this arm. - **W `tail`.** I count 260 out of 268, that is, except for six excluded for `image/svg+xml` and two duplicates; there is not a single `parse_error` in this arm, because the window at the end of the file is scanned and not finding a block is a measured „no“, not a failure. - **S `s_full`.** I count 252 out of 300, that is, except for one excluded for `image/gif`, 45 with `parse_error` and two duplicates. - **S `s_thumb`.** I count 259 out of 300, that is, except for one excluded for `image/gif`, 38 with `parse_error` and two duplicates. - **S `s_remote`.** I count 245 out of 296, that is, except for five failed fetches, three excluded (`image/gif`, `image/jpg` and one row without `Content-Type`), 41 with `parse_error` and two duplicates. - **The paired difference.** I count 242 pairs where both arms passed the content-type check and the parsing check and neither has `display_is_original`; out of 300 files, 30 such pairs dropped out, and 28 more for exclusion or a `parse_error` on one side.