← All work

Data collection engineering

Rental listings

Turning listing pages into structured records.

Custom data collection for a client

Python development · Data normalization · Integration testing

A listing through the pipeline

  1. DiscoveryFind source listing IDs
  2. Detail extractionRead available fields
  3. NormalizationMap values + check identity
  4. Integration boundaryApply write controls

Normalized listing fields

Illustrative records
nameproperty_typeaccommodatesbedroomsbathroomscitycountry
Harbor apartmentApartment421Example BayUnited States
Garden cottageCottage632.5Example BayUnited States
Hillside houseHouse8null3nullUnited States

Scroll to see all fields.

Real client identity withheld. Illustrative records with invented values, using fields supported by the collector. A null value represents missing source information.

The data shape

Consistent fields. Explicit gaps.

Replacement collection code for a property monitoring platform, covering listing discovery, detail extraction, record normalization, and an integration layer for the existing schema. The work includes testing against live sources.

The normalized record keeps property details in a defined shape. Counts and location fields can be missing; the output preserves that distinction for downstream handling.

Collection pipeline

From source pages to a defined record.

Collection and integration boundaries
Rental listing collection pipelineSource search and detail pages are retrieved through ZenRows. Python and Django collectors coordinated by Celery separate discovery from detail extraction. Records are normalized and passed to an integration layer with controlled writes into the existing schema. PostgreSQL and Sentry provide run evidence and diagnostics. Retrieve Parse Normalize Map Rental listing sourceSearch resultsProperty detail pagesVrbo ZenRowsSource-page retrievalShared fetch layer Python / Django collectorsDiscovery by geographic tile + paginationSeparate detail extractionCelery task coordination Normalized recordsDefined property fieldsSource-scoped identityExplicit missing values Existing platform schemaIntegration layer + controlled writesIdentity and collision checksIntegration boundary RUN EVIDENCEPostgreSQL · Run and attempt records Sentry · Diagnostic context

Retrieve source pages, identify listings, extract their fields, then normalize records for the existing platform schema.

Providers and responsibilities

  1. Listing source · Vrbo

    Search results identify listings; detail pages supply the property fields available from the source.

  2. ZenRows

    The fetch layer retrieves source pages for the collection code.

  3. Python / Django / Celery

    Geographic tiles and pagination organize discovery. A separate detail step extracts the fields for each listing.

  4. Normalized records

    Source-scoped identifiers and defined fields give the integration layer a consistent input. Missing values remain explicit.

  5. Existing platform schema

    An integration layer and write controls define how collected records can update the existing product.

  6. Run evidence

    PostgreSQL records collection runs and attempts. Sentry supplies diagnostic context for errors.

ZenRows handles source-page retrieval. Python collectors separate discovery from detail parsing, then map normalized fields to the existing platform schema through a controlled write boundary.View the full diagram →

Engineering decisions

Collection that can be inspected and checked.

Separate discovery from detail.

Geographic tiles and pagination organize the search for listing IDs. Detail extraction is a separate step, so finding a listing and reading its fields can be checked independently.

Make record identity explicit.

Identifiers are scoped by source, with collision checks at the integration boundary. Normalization produces defined fields while preserving missing values instead of substituting zero or an invented address.

Record what each run did.

Run and attempt records capture coverage, parsing outcomes, request cost, and latency. Explicit write controls separate collection from changes to the existing platform records.

View all work

Contact

What would you like to build or improve?

hello@outermost.us