o qjg @sdZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl mZddlmZddlmZmZddlmZddlmZddlZddlZddlZddlmZmZdd lm Z dd l!m"Z"m#Z#m$Z$dd l%m&Z&m'Z'dd l(m)Z)dd l*m+Z+m,Z,e ddl-m.Z.ddl/m0Z0ddl1m2Z2m3Z3m4Z4ddl5m6Z6ddl7m8Z8m9Z9m:Z:m;Z;mZ>m?Z?ddl@mAZAejBCddZDejBCddZEejBCddZFejBCddZGejBCddZHeIejBCddZJeIejBCd d!ZKeIejBCd"d#ZLeIejBCd$dZMd%eNfd&d'ZOd(ePd%dfd)d*ZQejBCd+d,RSZTejBCd-d.ZUejBCd/d0Vd1ZWejBCd2d3ZXejBCd4d5YZZeIejBCd6d7Z[eIejBCd8d9Z\eeXj]j^d:d:d;e_d<Z`e`aeZd=e`_bejcd>d?d@Zde`jeseeXe[e\dAdBZfefgede`hefeiZjejgede`hejejBCdCdDZkejBCdEdFZlejBCdGdZmeIejBCdHdIZnekekeldJZodKePd%ePfdLdMZpdNdOgZqe#dPdQdRZreTdSkreeUj^d:d:d;erjsdTe)eUdUdVdWddXl!m?Z?dYdZejBCd[dtd\DZud]e?d%dfd^d_ZvejBCd`dZweAdad=dbZxe=exfdcePd%dfdddeZyGdfdgdge+ZzGdhdidie+Z{djePd%ePfdkdlZ|dme6d%e9fdndoZ}dpdqZ~dred%ejfdsdtZdredued%eefdvdwZdxeedyed%efdzd{Zdredued|ePd%eefd}d~ZdePd%ePfddZdePd%ePfddZdePd%eePePePePffddZdePded%dfddZdedePdePdePd%ePf ddZdedePd%ePfddZdePded%dfddZdePdKePdePdePdePd%df ddZdePdKePdePdePdePd%df ddZerjde{e>eye>evgddezde"d]e?fddZerjde{e>eye>evgddezde"d]e?fddZerjde>eye>evgddde,d]e?deePfddZerjde>eye>evgddde,d]e?deePfddZerCdddZdS)u CD Report Generator API ======================== Single-file FastAPI service that: 1. Downloads a CD-entries CSV from a given URL 2. Generates one styled landscape-A3 PDF per drug (register name group) 3. Zips all the PDFs together 4. Uploads the zip to a target S3 "report_files/" location (built from a template in .env, using values parsed out of the CSV URL) 5. Returns immediately, then POSTs the result to the correct Laravel endpoint (stage or live) once background processing finishes. No database, no ORM models — everything happens in-memory / temp-dir per request. ----------------------------------------------------------------------------- CONFIG (env vars) ----------------------------------------------------------------------------- AWS_ACCESS_KEY_ID - AWS access key (or use an attached IAM role / profile) AWS_SECRET_ACCESS_KEY - AWS secret key AWS_REGION - e.g. "eu-west-2" (default: eu-west-2) S3_BUCKET - bucket name that backs the pharm.amazonaws.com domain S3_PUBLIC_BASE_URL - optional. Public base URL to build the returned link, e.g. "https://pharm.amazonaws.com". If not set, a presigned URL is generated instead. ZIP_KEY_TEMPLATE - template used to build the destination S3 key for the zip, using placeholders parsed from csv_url: {receive_form} - "local" / "stage" / "live" segment {pharmacy_id} - the folder segment right before "report_files" in csv_url {report_stem} - the CSV filename without ".csv" Default: "{receive_form}/pharmacies/{pharmacy_id}/report_files/{report_stem}.zip" STORAGE_BACKEND - "s3" (default) or "local". Set to "local" for local development/testing with NO AWS account or credentials needed at all — /generate-report will save the zip to LOCAL_OUTPUT_DIR on disk instead of uploading to S3. LOCAL_OUTPUT_DIR - folder to save zips into when STORAGE_BACKEND=local (default: "./local_zip_output") MAX_ROWS_PER_PDF - max rows rendered into ONE ReportLab document before generate_pdfs()/generate_single_pdf() split off another "_PartN" PDF. Exists to bound peak memory on huge CSVs — see the comment above the constant for the full explanation. Default: 20000 ----------------------------------------------------------------------------- CALLBACK ROUTING (Laravel) ----------------------------------------------------------------------------- csv_url's first path segment tells us which environment the request came from ("local", "stage", or "live"), which decides which Laravel endpoint gets the completion callback: csv_url segment -> callback goes to ---------------------------------------- local -> STAGE_LARAVEL_URL (no separate local Laravel API) stage -> STAGE_LARAVEL_URL live -> LIVE_LARAVEL_URL STAGE_LARAVEL_URL - default "https://devstage.pharmsmart.co.uk/api/ext/v2/pharmacy/aws-csv-pdf-converter/index.php" LIVE_LARAVEL_URL - default "https://pharmsmart.co.uk/api/ext/v2/pharmacy/aws-csv-pdf-converter/index.php" LARAVEL_SECRET_KEY - shared secret used to HMAC-sign the callback (Bearer token sent is HMAC-SHA256(pharmacy_id|csv_filename), hex digest). CALLBACK_TIMEOUT_SECONDS - request timeout for the callback POST (default: 30) ----------------------------------------------------------------------------- LOGGING ----------------------------------------------------------------------------- Every call to /generate-report, /generate-csv-to-pdf-report, and their /download variants (request received, validation result, background job progress, storage result, callback attempt/response, and any errors) is written to a rotating log file, in addition to stdout (so it still shows up under uvicorn/systemd/docker logs as before). LOG_FILE_PATH - path to the log file (default: "logs/api.log"). Parent directories are created automatically if missing. LOG_LEVEL - default "INFO" LOG_MAX_BYTES - size in bytes before the log file rotates (default: 5MB) LOG_BACKUP_COUNT- number of rotated backups to keep (default: 5) ----------------------------------------------------------------------------- RUN ----------------------------------------------------------------------------- pip install fastapi uvicorn pandas reportlab boto3 requests python-multipart python-dotenv Create a ".env" file in the same directory (see .env.example) then run: uvicorn main:app --host 0.0.0.0 --port 8000 ----------------------------------------------------------------------------- USAGE ----------------------------------------------------------------------------- POST /generate-report { "csv_url": "https://pharmsmart-pharmacy.s3.eu-west-2.amazonaws.com/local/pharmacies/N0tZbTJJRTFSYVhHanpGSm1EbVFKdz09/report_files/6a38ce059932a-cd-entries.csv" } Returns IMMEDIATELY: { "status": "success" } The heavy CSV -> PDFs -> zip -> upload work runs in a background task. Once it finishes, this JSON is POSTed to the Laravel endpoint chosen by the csv_url's environment segment (see CALLBACK ROUTING above): { "request_type": "csv-to-pdf", "timestamp": 1784021199, "pharmacy_id": "N0tZbTJJRTFSYVhHanpGSm1EbVFKdz09", "result_filename": "6a38ce059932a-cd-entries.zip", "csv_filename": "6a38ce059932a-cd-entries.csv" } with header: Authorization: Bearer On failure, "status": "failed" and "error": "" are added to that same payload (Laravel isn't required to look at these, but they're there for visibility / debugging). There is also POST /generate-report/download which streams the zip straight back in the HTTP response instead of uploading anywhere (handy for testing without S3 credentials). This one is still synchronous. POST /generate-csv-to-pdf-report and /generate-csv-to-pdf-report/download behave the same way but produce a single combined PDF (or, for CSVs larger than MAX_ROWS_PER_PDF rows, several PDF parts bundled into a zip — see MAX_ROWS_PER_PDF above) instead of one PDF per drug. N)escape)RotatingFileHandler)Path PurePosixPath)Optional)urlparse) BotoCoreError ClientError) load_dotenv)BackgroundTasksFastAPI HTTPException) FileResponse JSONResponse) StaticFiles) BaseModelHttpUrl)colors)TA_LEFT)A2A3 landscape)ParagraphStyle) LongTable ParagraphSimpleDocTemplateTable TableStyle)r r SecurityDependsRequest) APIKeyHeader AWS_REGIONz eu-west-2 S3_BUCKETS3_PUBLIC_BASE_URLZIP_KEY_TEMPLATEzF{receive_form}/pharmacies/{pharmacy_id}/report_files/{report_stem}.zipPDF_KEY_TEMPLATEzF{receive_form}/pharmacies/{pharmacy_id}/report_files/{report_stem}.pdfPRESIGNED_URL_EXPIRY_SECONDSiMAX_ROWS_PER_PDFi NMAX_REGISTER_GROUPSi MAX_MEMORY_MBreturncCsttjjdS)uCurrent process peak RSS in MB. ru_maxrss is KB on Linux (bytes on macOS) — this service is assumed to run on Linux servers.i)resource getrusage RUSAGE_SELF ru_maxrssr1r1(/var/www/html/pharmsmart-cdr-gen/main.py_current_rss_mbsr3contextc Cs@tdkrdSt}|tkrtdd|d|ddtdd dS) zyRaise a controlled, catchable error if RSS has crossed MAX_MEMORY_MB. No-op if MAX_MEMORY_MB is 0 (default/disabled).rNiz Aborting z: process RSS .0fzMB exceeded MAX_MEMORY_MB=zMB safety limit. status_codedetail)r+r3r )r4rssr1r1r2_check_memory_guardsr:STORAGE_BACKENDs3LOCAL_OUTPUT_DIRz./local_zip_outputLOCAL_PUBLIC_BASE_URLzhttp://localhost:8000/ LOG_FILE_PATHz logs/api.log LOG_LEVELINFO LOG_MAX_BYTESiPLOG_BACKUP_COUNTTparentsexist_ok cd_report_apiFz+%(asctime)s | %(levelname)-8s | %(message)sz%Y-%m-%d %H:%M:%S)fmtdatefmtutf-8)maxBytes backupCountencodingSTAGE_LARAVEL_URLzUhttps://devstage.pharmsmart.co.uk/api/ext/v2/pharmacy/aws-csv-pdf-converter/index.phpLIVE_LARAVEL_URLzLhttps://pharmsmart.co.uk/api/ext/v2/pharmacy/aws-csv-pdf-converter/index.phpLARAVEL_SECRET_KEYCALLBACK_TIMEOUT_SECONDS)localstagelive receive_formcCs6t|}|stdd|ddtdd|S)NzUnrecognised environment 'z' in csv_url (expected one of: z, z).r6)CALLBACK_URL_BY_ENVgetlowerr join)rXurlr1r1r2resolve_callback_url2sr_IDzAmended Entry IDzPharmsmart CD Report Generatorz1.0.0)titleversionrUz/files) directoryfilesname)r cCs"g|] }|r|dqS)r?)striprstrip).0or1r1r2 Ms  rkALLOWED_ORIGINS,requestcsrtsdS|jdp|jdpddsdStfddtDs7tddtd d d d dS) Noriginrefererr$r?c3s&|]}|kp|dVqdS)r?N) startswith)riallowedror1r2 `s$z verify_domain..zBlocked request: origin 'z' not in ALLOWED_ORIGINSizOrigin 'z' not allowed.r6)rlheadersr[rhanyloggerwarningr )rnr1rsr2 verify_domainTs"ryAPI_KEYz X-API-Key)rf auto_errorapi_keycCs6tstddd|r|tkrtdtddddS)NzServer API key not configured.r6z.Rejected request: invalid or missing X-API-KeyizInvalid or missing API key.)rzr rwrx)r|r1r1r2verify_api_keyls     r~c@eZdZUeed<dS)GenerateReportRequestcsv_urlN)__name__ __module__ __qualname__r__annotations__r1r1r1r2rw rc@r)GenerateReportResponsestatusN)rrrstrrr1r1r1r2r{rrrfcCs8t|}tdd|}tdd|}|ddpdS)Nz [^\w\s.-]r$z\s+__ Unknown_Drug)rrgresubrer1r1r2clean_filenames rstylecCstt|dd}t||S)N z
)rrreplacer)valuertextr1r1r2make_paragraphs rcsdg}|D]}t|}d|vrd|vr|dqd|vs$d|vr*|dqd|vr4|dqd|vs.)rrgr\appendrrsum)columnsweightsclc page_widthrr1rr2get_column_widthss<             rcsv_pathcs"gd}d}|D]}z tj||tdWSttfy+}z|}WYd}~qd}~wtjjy}zOz5gdttddffdd }tj||dt|d }r_t d t d |j d |WWYd}~St y}z |}WYd}~WYd}~qd}~wwd}~wwt d d|d)uRead a CSV trying multiple encodings, since pharmacy exports are often not UTF-8 (commonly Windows-1252 / Latin-1 from Excel/Windows systems). Also falls back to a more lenient parser if some rows have a ragged number of fields (e.g. an unescaped comma inside a text field). Reads every column as dtype=str: this data ends up rendered as strings anyway (callers already do .astype(str)), and reading it that way from the start avoids pandas' per-chunk numeric-type inference/conversion — which is what throws the 'Columns (...) have mixed types' DtypeWarning and roughly doubles peak memory during parsing on large files.)z utf-8-sigrLcp1252zlatin-1N)rOdtypebad_liner,cstdSN)rlen)rbad_rowsr1r2 _log_bad_linesz-read_csv_with_fallback.._log_bad_linepython)rOenginer on_bad_lineszSkipped z malformed row(s) in z? (ragged field count, e.g. an unescaped comma in a text field).rYzCould not parse CSV file: r6)pdread_csvrUnicodeDecodeError UnicodeErrorerrors ParserErrorlistrwrxrrf Exceptionr )rencodings_to_try last_errorrOerdfe2r1rr2read_csv_with_fallbacksB  r output_dirc st|dtjfddtDdd}jD]}d|vr.d|vr.|}nq|dur8jd}t}|j d d }t d |d |d |dt dd |t kr~t d|d|dt d|d tdd|d|dt ddtdddddd d!}td"dd#d$dd%d&td'd(d)d$d%d*tj}t|}g} d} j|d d+d,D]!\} } t| pd-} t| }tdt|t}|dkrt d.| d|d/|d0td1 t|D]}|t}t|t|}| j||}|dkrd2|dnd}|d3t| |d4}fd5d|Dg}|D]\}}|fd6d|Dq!d7t | }|dkrN|d8|dd9|d:7}t!t|t"t#ddd;dd<}t$||dd+d=}|%t&d>d?d@t'(dAfdBd?d@t'j)fdCd?dDdEt'j)fdFdGdHdIdJgt*|||g}|+|| |~~~~~t,-q~ | d7} | dKdks| |krt dL| dM|dNt| dOt dd t.dP| dM|d:qt dQt| dRtdS| S)TuGenerate one drug-wise styled PDF per group, returns list of PDF paths. Any register-name group larger than MAX_ROWS_PER_PDF rows is split into multiple "_PartN" PDFs, built and flushed to disk one at a time. This is what keeps peak memory bounded on very large CSVs (previously, a single huge group meant building millions of ReportLab Paragraph objects in memory at once before doc.build() ran — enough to get the whole worker process OOM-killed silently, with no exception ever logged).r$csg|] }|jvr|qSr1r)ricol)rr1r2rksz!generate_pdfs..rNrrfrF)dropnaz[generate_pdfs] CSV has z row(s), grouping by column 'z' -> z unique group(s) | RSS=r5MBz[generate_pdfs] Column 'z' has z. unique values, exceeding MAX_REGISTER_GROUPS=u. This almost always means the wrong column was picked as the drug/register-name grouping column (it should have low cardinality — a handful to a few hundred distinct drug names, not thousands of near-unique values). Refusing to generate u2 separate PDFs — check the CSV's column headers.rYzColumn 'z& unique values (> MAX_REGISTER_GROUPS=zy); refusing to generate that many per-drug PDFs. Check that the CSV's register/drug-name column was identified correctly.r6 ReportTitlez Times-Bold )fontNamefontSizeleading alignment spaceAfterHeaderg@gffffff @CJKrrrrwordWrapBodyz Times-Romang@)rrrrT)rsortz Unknown Drugz[generate_pdfs] Register ' rows -> splitting into  PDF part(s) of up to rows each_Part CD_Report_.pdfcg|]}t|qSr1rrir header_styler1r2rkAcrr1r)rir body_styler1r2rkCrz CD Reports - z (Part z of )pagesize leftMargin rightMargin topMargin bottomMargin) colWidths repeatRows splitByRow BACKGROUNDrrrz#D0D0D0 TEXTCOLORGRIDrrg?VALIGNrrTOP LEFTPADDINGrr RIGHTPADDINGrrr) TOPPADDINGrr) BOTTOMPADDINGrrr dz[generate_pdfs] Progress: r?z group(s), z PDF(s) so far | RSS=zgenerate_pdfs (after group z[generate_pdfs] Done:  PDF file(s) from total row(s))/rfillnaastyperdropEXCLUDE_COLUMNSrr\rnuniquerwinfor3r*errorr rrrgroupbyrgmaxmathceilr)rangeminilocriterrowsrtolistrrrrrsetStylerrHexColorblackrbuildgcrr:)rr register_colrtotal_rows_all unique_groups title_styler col_widths pdf_pathsgroups_processed drug_namegroup drug_label total_rows num_partspart_idxstartendchunksuffixpdf_path table_datarrow title_textdoctablestoryr1)rrrr2 generate_pdfss               r;r(zip_pathcCsPt|dtj}|D] }|j||jdq Wd|S1s!wY|S)Nr)arcname)zipfileZipFile ZIP_DEFLATEDwriterf)r(r<zipfpdfr1r1r2zip_pdfss rD base_filenamec st|dt}t|j}|dkrttntt}|d}d}d}|||t ddddt d d t d d d g} |jD] } t t| t|rR|| j nt| } | t | dq?t| pedfdd| D} t| } | kr| fdd| D} t|j}t|}t dt|t}|dkrtd|d|dtdg}t|D]}|t}t |t|}|j||}|dkrd|dnd}|||d}fdd|Dg}|D]\}}| fdd|Dqt|d| dd}|tdddtd fd!dd"d#tjfd$d%d&d'd(d)gtt||||ddd*}||g| |~~~~t !td+|dd,|d-t"d.d/t#d0|dd,|d1qtd2t|d3|d4|S)5urConvert an entire CSV into one or more compact landscape PDFs (no drug/register grouping — that's what generate_pdfs() does for /generate-report). Used by /generate-csv-to-pdf-report. Returns a list of PDF paths: a single-element list for CSVs up to MAX_ROWS_PER_PDF rows, or multiple "{base_filename}_PartN.pdf" files for larger ones — callers should zip the result together when more than one path comes back. Splitting (and freeing each chunk's memory before starting the next) is what keeps peak RAM bounded on huge CSVs instead of holding millions of ReportLab Paragraph objects in memory at once, which is what silently OOM-kills the worker process on very large files. Page size auto-scales from A3 to A2 for wide CSVs, and column widths are sized from actual content length so text wraps sensibly instead of overflowing.r$ r SinglePdfBody HelveticarErrSinglePdfHeaderHelvetica-Bold)parentr(rcsg|] }td|qS)#)r)ril) total_lenrr1r2rksz'generate_single_pdf..csg|]}|qSr1r1r)scaler1r2rksz[generate_single_pdf] rrrrrcrr1rrrr1r2rkrcrr1r)rivrr1r2rkrT)rrrrrrz#D9EAF7rrg?)FONTNAMErrrLrrr)rrrr)r rrrrz%[generate_single_pdf] Progress: part r?z done | RSS=r5rz generate_single_pdf (after part rz[generate_single_pdf] Done: r r )$rrrrrrrrrrrrrrrrrrr)rwrrrrrrrrrgreyrr!r"rr3r:)rrrErnum_cols page_sizer left_margin right_marginlengthsrmax_lenr' total_widthrr-r.r(r/r0r1r2r3r4r5rr6r9r8r1)rrrRrQrr2generate_single_pdfs   ,     r]rcCztt|jj}t|dkrtddd|d}|ddks$|dd kr*tdd d|d }t|d j}tj|||d S)z Extract: receive_form : local/stage/live pharmacy_id report_stem Example: https://.../local/pharmacies//report_files/.csv rJrYInvalid csv_url format.r6rr pharmacies report_filesUnexpected csv_url structure.r rErX pharmacy_id report_stem) rrpathpartsrr stemr&formatrrhrXrerfr1r1r2build_zip_key_from_csv_urls   rlcCr^)uSame as build_zip_key_from_csv_url but for the single-PDF endpoint — stores the PDF alongside the CSV in the same report_files/ folder.rJrYr_r6rrr`rarbrcr rErd) rrrgrhrr rir'rjrkr1r1r2build_pdf_key_from_csv_url!s   rmcCstt|jj}t|dks|ddks|ddkr tddd|d }|d }tjt|j}tj|d d }t |||||fS) alCheap, synchronous parse of csv_url -> (receive_form, pharmacy_id, csv_filename, zip_filename). receive_form is "local" / "stage" / "live" and decides which Laravel endpoint the completion callback goes to. Used so /generate-report can validate + build the response instantly, before the actual heavy work is handed off to the background task.rJrr`rarbrYrcr6rr r.zip) rrrgrhrr osbasenamesplitextr_)rrhrXre csv_filename zip_filenamer1r1r2parse_csv_url_parts8s$  rt dest_pathc CszMtj|ddd:}|t|d}|jddD]}||qWdn1s,wYWdWdSWdWdS1sFwYWdStjyc}z tdd|d d}~ww) NT<)streamtimeoutwbi) chunk_sizeizFailed to download CSV: r6)requestsr[raise_for_statusopen iter_contentrARequestExceptionr )rrurfr2rr1r1r2 download_csvQs  &rlocal_zip_pathbucketkeykey_flagc Cs|stdddtjdtd}z/|dkr"|jt|||ddid n|d kr4|jt|||dd id n |t|||WnttfyX}z tdd |d |dd}~wwtrht dd| dSz |j d||dt dWSttfy}z tdd|dd}~ww)Nr}zONo S3 bucket configured. Set S3_BUCKET env var or pass 'bucket' in the request.r6r<) region_namezip_file ContentTypeapplication/zip) ExtraArgspdf_fileapplication/pdfzFailed to upload z to S3: r? get_object)BucketKey)Params ExpiresInz(Zip uploaded but failed to build a URL: ) r boto3clientr" upload_filerrr r%rhlstripgenerate_presigned_urlr()rrrrr<rr1r1r2 upload_to_s3\s:rc Csltt|}|jjdddzt||Wnty+}z tdd|dd}~wwtd| dS) zGLocal-disk equivalent of upload_to_s3, used when STORAGE_BACKEND=local.TrFr}zFailed to save zip locally: r6Nz/files/r?) rr=rMmkdirshutilcopyfileOSErrorr r>r)rrrurr1r1r2 save_to_local{s r callback_urlpayloadc Cs.|d}|d}|r|stddS|d|}ttd|dtj }dd|d }t d |d |d |d |ddz.t j |||t d}t d|d|j|jsutd|d|jddWdSWdSt jy}ztd|d|WYd}~dSd}~ww)aPOST the final result to the resolved Laravel endpoint, with an HMAC-signed Bearer token. Best-effort: logs a warning on failure instead of raising, since we're already inside a fire-and-forget background task at this point. The token itself is never logged.rerrzECannot send callback: missing pharmacy_id or csv_filename in payload.N|rLzapplication/jsonzBearer )z Content-Type AuthorizationzSending callback -> z | pharmacy_id= csv_filename=z status=rsuccess)jsonrurxzCallback response <- z | http_status=zCallback non-OK response from : r}zCallback request FAILED to )r[rwrxhmacnewAWS_FILE_CONVERT_API_KEYencodehashlibsha256 hexdigestrr{postrSr7okrrr) rrrefilenamedatatokenruresprr1r1r2 send_callbacksL    &$rrerrrsc Cst|}ttjdd}d|d|d|}td|dtt|||d}z%z|d } |d } | t || td | j d |t | | } | sst d |t|i|dddWWtj|dddStdt| d|t|} |t| j} t| | tdt| d| jd|tdkrt| | }ntrtdkrd}t| t| |}nt| | }td|d|t||td|WnZty }z%t d|jd|jd |t|i|dt|jdWYd}~n7d}~wty3}ztd!|t|i|dt|dWYd}~nd}~wwWtj|dddSWtj|dddSWtj|dddStj|ddw)"a*Runs the actual CSV -> PDFs -> zip -> upload pipeline. Called via BackgroundTasks so /generate-report can return immediately. Always ends by POSTing a result (success or failure) to the Laravel URL resolved from receive_form (local/stage -> STAGE_LARAVEL_URL, live -> LIVE_LARAVEL_URL). cd_report_prefix pharmacy_id=r env=z+[generate-report] BACKGROUND JOB STARTED | csv-to-pdf request_typerreresult_filenamerr input.csvpdfsz"[generate-report] CSV downloaded ( bytes) | z3[generate-report] No PDFs generated (empty CSV?) | failed2No PDFs were generated from the CSV (empty file?).rrT ignore_errorsNz[generate-report] Generated z PDF(s) | z[generate-report] Zipped z PDF(s) ->  | rUr<rz[generate-report] Zip saved -> z7[generate-report] BACKGROUND JOB COMPLETED (success) | z7[generate-report] BACKGROUND JOB FAILED (HTTPException r) | z=[generate-report] BACKGROUND JOB FAILED (unexpected error) | )r_rtempfilemkdtemprwrinttimerrstatst_sizer;rrrrmtreerrlrfrDr;rr#rr r7r8rr exception)rrXrerrrsrwork_dirlog_ctx base_payloadrpdf_dirr(zip_keyr<zip_urlrrr1r1r2process_report_backgrounds     & "      "r pdf_filenamec Cst|}ttjdd}d|d|d|}td|dtt|||d}zFz|d } |d } | t || td | j d |t j |d } t| | | } | s|td|t|i|dddWWtj|dddStdt| d|t| dkr| d } t|}|}d}n$| d}||} t| | t|}d}tdt| d|d|||d<tdkrt| |}ntrtd krt| t||}nt| |}td!|d|t||td"|WnZty,}z%td#|jd$|jd%|t|i|dt|jdWYd}~n7d}~wt yT}zt!d&|t|i|dt|dWYd}~nd}~wwWtj|dddSWtj|dddSWtj|dddStj|ddw)'uPRuns the CSV -> PDF(s) -> upload pipeline for /generate-csv-to-pdf-report. Same background-task / callback-routing pattern as process_report_background(), but produces one combined PDF via generate_single_pdf() — UNLESS the CSV has more than MAX_ROWS_PER_PDF rows, in which case generate_single_pdf() returns multiple "_PartN" PDFs and this function bundles them into a zip instead (same OOM-avoidance reasoning as process_report_background). Either way, the callback payload's 'result_filename' points at whichever file actually got uploaded (.pdf or .zip). csv_to_pdf_rrrrz6[generate-csv-to-pdf-report] BACKGROUND JOB STARTED | rrrrz-[generate-csv-to-pdf-report] CSV downloaded (rrz=[generate-csv-to-pdf-report] No PDF generated (empty CSV?) | r0No PDF was generated from the CSV (empty file?).rTrNz.[generate-csv-to-pdf-report] PDF generated -> z part(s) | rrrnrzB[generate-csv-to-pdf-report] CSV too large for one PDF -> bundled parts into rrrUr<z-[generate-csv-to-pdf-report] Result saved -> zB[generate-csv-to-pdf-report] BACKGROUND JOB COMPLETED (success) | zB[generate-csv-to-pdf-report] BACKGROUND JOB FAILED (HTTPException rrzH[generate-csv-to-pdf-report] BACKGROUND JOB FAILED (unexpected error) | )"r_rrrrwrrrrrrrrorgrqr]rrrrrrmrDrlr;rr#rr r7r8rrr)rrXrerrrrrrrrr base_namer( result_path result_keyrr result_urlrr1r1r2process_csv_to_pdf_background s    4         "rz/generate-report)response_model dependenciesreqbackground_tasksc Cst|j}|jr |jjnd}td|d|z t|\}}}}Wn tyC} ztd|d|d| j d| j d} ~ wwtd|d |d |d || t |||||td |d |t d dS)a;Validates csv_url, kicks off the CSV -> PDF -> zip -> upload pipeline in the background, and returns immediately. Once the background job finishes (success or failure), the result is POSTed to whichever Laravel endpoint matches the csv_url's environment segment (local/stage -> stage, live -> live).unknownz([generate-report] REQUEST RECEIVED from | csv_url=z([generate-report] REQUEST REJECTED from rrNz*[generate-report] REQUEST VALIDATED | env= pharmacy_id=rz zip_filename=z6[generate-report] Background job queued | pharmacy_id=rr)rrrhostrwrrtr rr7r8add_taskrr) rrrnr client_iprXrerrrsrr1r1r2generate_reportss>  rz/generate-csv-to-pdf-reportc Cst|j}|jr |jjnd}td|d|z t|\}}}}Wn tyC} ztd|d|d| j d| j d} ~ wwt j |dd } td |d |d |d | |t||||| td|d |tddS)u]Same request/response/callback-routing contract as /generate-report, but converts the whole CSV into ONE combined landscape PDF (via generate_single_pdf()) instead of one PDF per drug + a zip — unless the CSV is large enough to need splitting (see MAX_ROWS_PER_PDF), in which case the result is bundled into a zip instead, same as /generate-report. The result is stored alongside the CSV in the same report_files/ S3 (or local) folder. Request body is identical to /generate-report: {"csv_url": "https://.../local/pharmacies//report_files/.csv"} Returns IMMEDIATELY: {"status": "success"} Once the background job finishes, the result is POSTed to whichever Laravel endpoint matches the csv_url's environment segment (same local/stage -> stage, live -> live routing as /generate-report): { "request_type": "csv-to-pdf", "timestamp": 1784021199, "pharmacy_id": "N0tZbTJJRTFSYVhHanpGSm1EbVFKdz09", "result_filename": "6a38ce059932a-cd-entries.pdf", "csv_filename": "6a38ce059932a-cd-entries.csv" } rz3[generate-csv-to-pdf-report] REQUEST RECEIVED from rz3[generate-csv-to-pdf-report] REQUEST REJECTED from rrNrrz5[generate-csv-to-pdf-report] REQUEST VALIDATED | env=rrz pdf_filename=zA[generate-csv-to-pdf-report] Background job queued | pharmacy_id=rr)rrrrrwrrtr rr7r8rorgrqrrr) rrrnrrrXrerr _zip_filenamerrr1r1r2generate_csv_to_pdf_reports@  rz/generate-report/download)rc Cs|jr|jjnd}td|d|ttjdd}|d}|d}|zrtt ||td| j d |t ||}|sSt d |td d d tdt|d||sytjtt |j}tj|dd}n |ds|d}||} t|| td|d|Wn2tytj|ddty} ztd|tj|ddtdd| d d} ~ wwtt | |ddS)uSame pipeline, but streams the zip back directly instead of uploading to S3. Useful for local testing without AWS credentials. This endpoint remains synchronous (no background task, no callback) since the point is to get the file back in the same request. zip_filename is OPTIONAL — if omitted, it's derived automatically from csv_url, e.g. ".../2906266-cd-entries.csv" -> "2906266-cd-entries.zip". rz1[generate-report/download] REQUEST RECEIVED from r cd_report_dl_rrrz+[generate-report/download] CSV downloaded ( bytes) | csv_url=zD[generate-report/download] No PDFs generated (empty CSV?) | csv_url=rYrr6z%[generate-report/download] Generated z PDF(s) | csv_url=rrnz;[generate-report/download] Streaming zip back to client -> TrzG[generate-report/download] REQUEST FAILED (unexpected error) | csv_url=r}Failed to generate report: Nrrgr media_type)rrrwrrrrrrrrrr;rr rrorgrprrqr\endswithrDrrrrr) rrnrsrrrrr(rrr<rr1r1r2generate_report_downloadsH      rz$/generate-csv-to-pdf-report/downloadc Cs|jr|jjnd}td|d|ttjdd}|d}|d}|ztt ||td| j d ||sWt j tt |j }t j |d d }n |d sc|d }t j |d }t|||} | std |td ddt| dkr| d } |} d} n |d} || } t| | d} tdt| d| d|td| d|Wn2tytj|ddty} ztd|tj|ddtdd| dd} ~ wwtt | | | dS)uRSame pipeline as /generate-csv-to-pdf-report, but streams the result back directly instead of uploading to S3 — the PDF equivalent of /generate-report/download. Returns a single .pdf for CSVs up to MAX_ROWS_PER_PDF rows, or a .zip of "_PartN" PDFs for larger ones (same fallback as the background version). Synchronous (no background task, no callback) since the point is to get the file back in the same request. pdf_filename is OPTIONAL — if omitted, it's derived automatically from csv_url, e.g. ".../2906266-cd-entries.csv" -> "2906266-cd-entries.pdf". rz<[generate-csv-to-pdf-report/download] REQUEST RECEIVED from rcsv_to_pdf_dl_rrrz6[generate-csv-to-pdf-report/download] CSV downloaded (rrrzN[generate-csv-to-pdf-report/download] No PDF generated (empty CSV?) | csv_url=rYrr6rrrnrzK[generate-csv-to-pdf-report/download] CSV too large for one PDF -> bundled rzI[generate-csv-to-pdf-report/download] Streaming result back to client -> TrzR[generate-csv-to-pdf-report/download] REQUEST FAILED (unexpected error) | csv_url=r}rNr)rrrwrrrrrrrrrrorgrprrqr\rr]rr rrDrrrrr)rrnrrrrrrrrr(rrrrr1r1r2#generate_csv_to_pdf_report_downloadsf        rz/healthcCs tddiS)Nrr)rr1r1r1r2healthZs rr)__doc__r"rrloggingrrorr-rrrr>htmlrlogging.handlersrpathlibrrtypingr urllib.parserrpandasrr{botocore.exceptionsrr dotenvr fastapir r r fastapi.responsesrrfastapi.staticfilesrpydanticrr reportlab.librreportlab.lib.enumsrreportlab.lib.pagesizesrrrreportlab.lib.stylesrreportlab.platypusrrrrrrrr fastapi.securityr!environr[r"r#r%r&r'rr(r)r*r+floatr3rr:rgr\r;r=rhr>r@upperrArCrDrMr getLoggerrwsetLevel propagate Formatter_log_formatterhandlers _file_handler setFormatter addHandler StreamHandler_console_handlerrPrQrrSrZr_rappmountsplitrlryrzapi_key_headerr~rrrrr DataFramerrr;rDr]rlrmtuplertrrrdictrrrrrrrrrr1r1r1r2s<                        &/y"  1 R j'>8F