Wiki source code of Compteur de téléchargements Python

Last modified by Mélodie on 2026/05/07 14:20

version	line-number	content
15.1	1	== Objective ==
1.1	2
15.1	3	Set up an automatic download counter for ISO files hosted on ##downloads.linuxvillage.org##, with:
1.1	4
15.1	5	* Daily parsing of Apache logs
	6	* Deduplication by IP + file + day (to avoid false duplicates from HTTP 206 Partial Content requests in resumed downloads)
	7	* Month-by-month counter aggregation in a JSON file
	8	* Automatic generation of a public page listing available ISOs
	9	* Automatic generation of a private monthly statistics page
1.1	10
15.1	11	== Prerequisites ==
1.1	12
15.1	13	* Python 3 installed (##/usr/bin/python3##)
	14	* Root access to the server
	15	* Apache logs available at ##/var/log/apache2/downloads-ssl-access.log*##
	16	* ##cron## package installed: ##apt install cron##
1.1	17
15.1	18	== Script Installation ==
1.1	19
15.1	20	Create the directory and place the script there (attachment to this page, to be renamed without the final ##.txt##):
1.1	21
	22	{{code language="bash"}}
	23	mkdir -p /opt/download_stats
15.1	24	# Place download_stats.py in this directory
1.1	25	chmod 750 /opt/download_stats/download_stats.py
	26	{{/code}}
	27
15.1	28	== .stats Directory Structure ==
1.1	29
15.1	30	The script automatically creates and manages the hidden ##.stats## directory at the root of the download space:
1.1	31
	32	{{code language="bash"}}
	33	mkdir /var/www/file-server/.stats
	34	{{/code}}
	35
15.1	36	This directory will contain:
1.1	37
15.1	38	* ##counts.json## : cumulative history of monthly counters
	39	* ##index.html## : private statistics page
1.1	40
15.1	41	The statistics page is accessible at the URL:
1.1	42	##https:~/~/downloads.linuxvillage.org/.stats/##
	43
15.1	44	This URL is accessible but not publicly referenced.
1.1	45
15.1	46	== Cron Configuration ==
1.1	47
15.1	48	As root:
1.1	49
	50	{{code language="bash"}}
	51	crontab -e
	52	{{/code}}
	53
15.1	54	Add the following line (daily execution at 01:00 UTC):
1.1	55
	56	{{code language="bash"}}
	57	0 1 * * * /usr/bin/python3 /opt/download_stats/download_stats.py
	58	{{/code}}
	59
15.1	60	== JSON Reset ==
1.1	61
15.1	62	If needed (structure change, counter reset):
1.1	63
	64	{{code language="bash"}}
	65	echo '{}' > /var/www/file-server/.stats/counts.json
	66	/usr/bin/python3 /opt/download_stats/download_stats.py
	67	{{/code}}
	68
15.1	69	== Generated Pages ==
1.1	70
15.1	71	The script generates two HTML pages on each run:
1.1	72
15.1	73	; Public page
	74	: ##/var/www/file-server/index.html## → lists available ISOs with their size, upload date, and links to checksums (.md5, .sha512). Content is automatically detected by directory scanning. The page follows the graphic identity of https://linuxvillage.org (colors, fonts, layout).
1.1	75
15.1	76	; Statistics page (private)
	77	: ##/var/www/file-server/.stats/index.html## → table of downloads month by month, with per-file totals and grand total.
1.1	78
15.1	79	== Script Logic ==
1.1	80
15.1	81	; Log parsing
	82	: The script scans all ##downloads-ssl-access.log*## files (including gzipped files from log rotation). Only GET and HEAD requests for ##.iso## files with HTTP status codes 200 or 206 are processed.
1.1	83
15.1	84	; Deduplication
	85	: For each retained line, a key ##(ip, file, day)## is constructed. If this key has already been seen in the same run, the line is ignored. This prevents counting a resumed download multiple times when it generates multiple 206 requests.
1.1	86
15.1	87	; Monthly JSON structure
	88	: Counters are stored by file and by month (##YYYY-MM##). On each run, the script merges new counters with the historical data by taking the maximum of the two values → this preserves data prior to the log retention window (14 days) without creating double-counting.
1.1	89
15.1	90	; Directory scanning
	91	: The list of files displayed on the HTML pages is built dynamically on each run. Adding a new ISO version requires no manual intervention.
1.1	92
15.1	93	== Attachment ==
1.1	94
15.1	95	The ##download_stats_en.py.txt## file attached to this page is the source Python script.
	96	Rename to ##download_stats_en.py## after download, then:
1.1	97
	98	{{code language="bash"}}
15.1	99	chmod 750 /opt/download_stats/download_stats_en.py
1.1	100	{{/code}}