leaf_engine.io.api.crud_caller ============================== .. py:module:: leaf_engine.io.api.crud_caller Classes ------- .. autoapisummary:: leaf_engine.io.api.crud_caller.LeafCRUDCaller Module Contents --------------- .. py:class:: LeafCRUDCaller(endpoint: Optional[str] = None, key: Optional[str] = None, **kwargs) Bases: :py:obj:`leaf_engine.io.api.base_caller.LeafBaseAPICaller` Makes calls to the analytics CRUD API. .. py:method:: _check_responses(responses: List[requests.Response]) -> None Checks responses for errors (response.status_code >= 400). :param responses: List of responses to check. :type responses: List[requests.Response] :raises LeafAPIException: Raises if any response is not ok and prints the status code and response text. .. py:method:: call_rpc(function_name: str, **kwargs) -> requests.Response Calls an RPC function. :param function_name: Name of the function in the database to call as an RPC. :type function_name: str :param kwargs: A dictionary of arguments to pass to the RPC function. :type kwargs: Mapping[str, Any], optional :raises LeafAPIException: _description_ .. py:method:: check_existing_ids(uuids: List[str], table_name: str, company_id: int, batch_date: str, batch_size: int = 1000, verbose: int = 0) -> List[bool] Receives a list of UUIDs and returns a list of uuid's that already exist in the database. .. py:method:: count(path: str, **kwargs) -> int Counts the number of records in a table. :param path: Analytics table to count. :type path: str :param kwargs: Can be used to pass additional query parameters, for example `company_id="eq.17852"`. See PostgREST documentation for details. :raises LeafAPIException: Raises if count value is not returned. :returns: Number of records :rtype: Optional[int] .. py:method:: delete(path: str, **kwargs) -> requests.Response .. py:method:: delete_batch(company_id: int, batch_date: str) -> None Deletes all records associated with a batch for a given company. This includes: - raw_shipment - normalized ETL tables (location, lane, shipment) - explorer outputs - network moves - batch and datasets :param company_id: Company ID to match batch. :type company_id: int :param batch_date: Batch date to match batch. :type batch_date: str .. py:method:: delete_network_moves(company_id: int, **kwargs) -> requests.Response Deletes network moves for company_id. :param company_id: Delete moves for company. :type company_id: int :param kwargs: Can be used to pass additional filters on delete. .. py:method:: get(path: str, headers: Optional[Mapping[str, str]] = None, **kwargs) -> requests.Response Makes a GET request to CRUD API. :param path: Analytics table to GET. :type path: str :param headers: Additional headers to pass to API. See PostgREST documentation for details. :type headers: Optional[Mapping[str, str]], optional :param kwargs: Can be used to pass additional query parameters. See PostgREST documentation for details. :returns: Response object. :rtype: requests.Response .. py:method:: get_batch(company_id: int, batch_date: str, **kwargs) -> Optional[dict] Gets batch parameters for a given company and batch date. :param company_id: Company ID to get batch for. :type company_id: int :param batch_date: Batch date to get batch for. :type batch_date: str :returns: Batch parameters if batch exists, else None. :rtype: Optional[dict] :raises LeafAPIException: Raises if multiple batches are found. .. py:method:: get_cluster_schema(company_id: int, name: str, record_types: list[str], **kwargs) -> dict Gets cluster schema for a given company and name. :param company_id: Company ID to get cluster schema for. :type company_id: int :param name: Name to get cluster schema for. :type name: str :returns: Cluster schema. :rtype: dict .. py:method:: get_company_fuel_surcharge_schedules(company_id: int) -> pandas.DataFrame Queries DB for all company fuel surcharge schedules. :param company_id: Company ID to get applicable FSCs for. .. py:method:: get_company_network_lanes(company_id: int) -> pandas.DataFrame Gets network lanes for a given company. Network lanes are used to populate the lane graph. Check analogous materialized view in the analytics schema `analytics.mv_company_network_lanes` for more details. Note: the latest available batch date is used to get company network lanes. :param company_id: ID of the company to get network lanes for. Use -1 to get network lanes for all companies. :type company_id: int :raises LeafAPIException: If no network lanes are found for the given company. :returns: DataFrame of network lanes. Follows the same schema as in `notebooks/08 - network_lanes.ipynb`. :rtype: pd.DataFrame .. py:method:: get_corridors(**kwargs) Gets all corridors. .. py:method:: get_execution_shipments(**kwargs) Gets all execution shipments for building the execution graph. .. py:method:: get_lanes(company_id: int) -> pandas.DataFrame Gets all lanes. :returns: DataFrame of lanes :rtype: pd.DataFrame .. py:method:: get_lighthouse_lanes(company_ids: List[int]) -> pandas.DataFrame Gets lanes for lighthouse shippers. Used for Internal Type 2 Runs. See the materialized view in the migrations repository `analytics.mv_lighthouse_lanes` for more details. Note: the latest available batch date is used to get lanes. :param company_ids: List of company IDs to get lanes for. :type company_ids: List[int] :returns: DataFrame of lanes. :rtype: pd.DataFrame .. py:method:: get_locations(location_ids: Iterable[str], **kwargs) -> pandas.DataFrame Gets locations given a list of location ids. :param location_ids: A list of location ids to retrieve data for. :type location_ids: Iterable[str] :param kwargs: Can be used to pass additional parameters to :func:`leaf_engine.io.api.crud_caller.get`. :returns: DataFrame of locations :rtype: pd.DataFrame .. py:method:: get_many(path: str, headers: Optional[Mapping[str, str]] = None, batch_size: int = -1, n_jobs: int = -1, verbose: int = 0, **kwargs) -> List[requests.Response] Makes parallel GET requests to CRUD API. See :method:`leaf_engine.io.api.crud_caller.get` for details. .. py:method:: get_network_moves(company_id: int, record_types: list[str], **kwargs) -> pandas.DataFrame Gets network moves for a given company ID. :param company_id: Company ID to filter on. :type company_id: int :param record_types: List of record types to filter on. :type record_types: list[str] :returns: DataFrame of network moves. :rtype: pd.DataFrame .. py:method:: get_point_locations(company_id: int, record_types: list[str], **kwargs) -> pandas.DataFrame Gets all point locations. :param company_id: Company ID to filter on. :type company_id: int :param record_types: List of record types to filter on. :type record_types: list[str] :returns: DataFrame of point locations :rtype: pd.DataFrame .. py:method:: get_raw_shipments(company_id: int, batch_date: str, **kwargs) -> pandas.DataFrame Gets raw shipment records from the analytics.raw_shipments table. :param company_id: Company ID to get raw shipments for. :type company_id: int :param batch_date: Batch date to get raw shipments for. Must be in YYYY-MM-DD format. :type batch_date: str :returns: DataFrame of raw shipments :rtype: pd.DataFrame .. py:method:: normalize_raw_lanes(company_id: int, batch_date: str) -> None Updates the location, lane, and powerlane tables. Normalizes raw_lane table rows for a given company and batch date. The location table is updated with point and cluster locations. The lane table is updated with point-to-point and power lanes. :param company_id: Company ID to update lanes for. :type company_id: int :param batch_date: Batch date to update lanes for. :type batch_date: str .. py:method:: normalize_raw_shipments(company_id: int, batch_date: str) -> None Updates the location, lane, and shipment tables. Normalizes raw_shipment table rows for a given company and batch date. The location table is updated with point and cluster locations. The lane table is updated with point-to-point and power lanes. The shipment table is updated with shipments. :param company_id: Company ID to update shipments for. :type company_id: int :param batch_date: Batch date to update shipments for. :type batch_date: str .. py:method:: post(path: str, data: Union[Mapping[str, Any], List[Mapping[str, Any]], List[Dict[str, Any]]], headers: Optional[Mapping[str, str]] = None, return_columns: Optional[List[str]] = None, **kwargs) -> requests.Response Makes a POST request to CRUD API. :param path: Analytics table to write to. :type path: str :param data: Record or records to write. :type data: Union[Mapping[str, Any], List[Mapping[str, Any]]] :param headers: Additional headers to pass to API. See PostgREST documentation for details. :type headers: Optional[Mapping[str, str]], optional :param return_columns: List of columns to return in response, defaults to `["id"]`. See PostgREST documentation for details. :type return_columns: Optional[List[str]], optional :returns: Response object. :rtype: requests.Response .. py:method:: post_batch(batch_params: dict[str, Any]) -> requests.Response Creates a batch record in the database from a dictionary. Also creates associated records in the dataset, batch_dataset, and batch_equipment_class tables. :param batch_params: Batch parameters to insert. :type batch_params: Mapping[str, Any] .. py:method:: post_carrier_churn(df: pandas.DataFrame, batch_size: int = 100, **kwargs) -> List[requests.Response] Inserts carrier churn DataFrame into `carrier_churn` table. .. py:method:: post_cluster_schema(schema: dict, batch_date: str) -> dict Creates a cluster schema record in the database from a dictionary. :param schema: Cluster schema parameters to insert. :type schema: Mapping[str, Any] :returns: Cluster schema. :rtype: dict .. py:method:: post_consolidated_flex(df: pandas.DataFrame, batch_size: int = 100, **kwargs) -> List[requests.Response] Inserts consolidated flex DataFrame into `consolidated_flex` table. .. py:method:: post_lane_adapt_detail(records: List[Mapping[str, Any]], batch_size: int = 100, **kwargs) Adds benefit, actions, observations to analytics.lane_adapt. The analytics.lane_adapt table is used to populate the lane explorers. Updates existing records in place. .. py:method:: post_lane_quick_ref(df: pandas.DataFrame, batch_size: int = 100, **kwargs) -> List[requests.Response] Inserts lane quick reference DataFrame into `lane_quick_ref` table. .. py:method:: post_many(path: str, records: List[Mapping[str, Any]], headers: Optional[Mapping[str, str]] = None, return_columns: Optional[List[str]] = None, batch_size: int = -1, n_jobs: int = -1, verbose: int = 0, **kwargs) -> List[requests.Response] Makes a parallel POST requests to CRUD API. See :method:`leaf_engine.io.api.crud_caller.post` for details. .. py:method:: post_network_moves(df: pandas.DataFrame, batch_size: int = 100, **kwargs) -> List[requests.Response] Inserts network moves DataFrame into `network_move` table. :param df: Network moves `pd.DataFrame` with `company_id` and `batch_date` columns. :type df: pd.DataFrame :param batch_size: Inserts per call. Defaults to 1000. :type batch_size: int, optional :param kwargs: Can be used to pass additional parameters to :func:`leaf_engine.io.api.crud_caller.post_many`. :returns: List of responses. :rtype: List[requests.Response] .. py:method:: post_observations_patterns(df: pandas.DataFrame, batch_size: int = 100, **kwargs) -> List[requests.Response] Inserts observation patterns DataFrame into `observation_pattern` table. .. py:method:: post_raw_lanes(records: List[Mapping[str, Any]], batch_size: int = 100, **kwargs) -> List[requests.Response] Inserts raw lanes into the raw_lane table. :param records: List of records (dicts) of raw lanes to insert. :type records: List[Mapping[str, Any]] :param batch_size: Number of records to insert in each call. :type batch_size: int .. py:method:: post_raw_shipments(records: List[Mapping[str, Any]], batch_size: int = 100, **kwargs) -> List[requests.Response] Inserts raw shipments into the raw_shipment table. :param records: List of records (dicts) of raw shipments to insert. :type records: List[Mapping[str, Any]] :param batch_size: Number of records to insert in each call. :type batch_size: int .. py:method:: post_should_be_flex(df: pandas.DataFrame, batch_size: int = 100, **kwargs) -> List[requests.Response] Inserts should be flex DataFrame into `should_be_flex` table. .. py:method:: refresh_mvs() Calls the refresh_company_network_lanes and refresh_lighthouse_lanes functions to refresh the materialized views. .. py:method:: update_routing_tables(company_id: int, batch_date: str) -> None Updates routing tables for a given company and batch date. :param company_id: Company ID to update routing for. :type company_id: int :param batch_date: Batch date to update routing for. :type batch_date: str .. py:attribute:: endpoint .. py:attribute:: key