{ "cells": [ { "cell_type": "markdown", "id": "bf16a6eb-44a2-4dd7-8d4d-598e6f2c9b24", "metadata": {}, "source": [ "# Silver " ] }, { "cell_type": "markdown", "id": "4405f172-43b6-4245-bcd8-2c6e01fada3a", "metadata": {}, "source": [ "We can download the data from the CFTC via Python.\n", "\n", "Many data APIs become commercial for recent prices. " ] }, { "cell_type": "code", "execution_count": 1, "id": "f3704b41-1622-4596-ae99-3201b9bd8e14", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Selected: disaggregated_fut\n", "Downloaded single year data from: 2020\n", "Stored the file f_year.txt in the working directory.\n", "Selected: disaggregated_fut\n", "Downloaded single year data from: 2021\n", "Stored the file f_year.txt in the working directory.\n", "Selected: disaggregated_fut\n", "Downloaded single year data from: 2022\n", "Stored the file f_year.txt in the working directory.\n", "Selected: disaggregated_fut\n", "Downloaded single year data from: 2023\n", "Stored the file f_year.txt in the working directory.\n", "Selected: disaggregated_fut\n", "Downloaded single year data from: 2024\n", "Stored the file f_year.txt in the working directory.\n" ] } ], "source": [ "import pandas as pd\n", "import cot_reports as cot # Ensure cot_reports is correctly imported and cot.cot_year() works as expected\n", "\n", "df = pd.DataFrame()\n", "begin_year = 2020\n", "end_year = 2024\n", "\n", "for i in range(begin_year, end_year + 1):\n", " # Assuming cot.cot_year returns a DataFrame\n", " single_year = pd.DataFrame(cot.cot_year(i, cot_report_type='disaggregated_fut'))\n", " single_year.to_csv(f'./COT_CFTC_{i}.csv', index=False)" ] }, { "cell_type": "code", "execution_count": 2, "id": "f4f90eb0-dca4-4589-a385-408c66c6e0a0", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['COT_CFTC_2022.csv', 'COT_CFTC_2023.csv', 'COT_CFTC_2021.csv', 'COT_CFTC_2020.csv', 'COT_CFTC_2024.csv']\n" ] } ], "source": [ "import glob\n", "\n", "# Adjust the path and pattern according to your CSV files location and naming convention\n", "csv_files = glob.glob('COT_CFTC_20*.csv')\n", "print(csv_files)" ] }, { "cell_type": "markdown", "id": "414b7248-5acc-46a9-916a-cbed7228331f", "metadata": {}, "source": [ "At this point, we have the CFTC Disaggregated Futures report from 2022 to 2024 (until today)" ] }, { "cell_type": "markdown", "id": "084eb98b-1914-49e0-b57a-4e7717ea159c", "metadata": {}, "source": [ "## Load the data, and prepare the dataset\n", "\n", "The CFTC report contains data on all futures, not just silver." ] }, { "cell_type": "code", "execution_count": 3, "id": "cb167036-4f9b-4405-a942-71fae7d7aec7", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/tmp/ipykernel_1629470/490941526.py:1: DtypeWarning: Columns (120,121,132,133,134,135,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160) have mixed types. Specify dtype option on import or set low_memory=False.\n", " df = pd.read_csv('COT_CFTC_2024.csv')\n" ] } ], "source": [ "df = pd.read_csv('COT_CFTC_2024.csv')" ] }, { "cell_type": "code", "execution_count": 4, "id": "2f554d5a-e6da-4a6c-a16e-f2b3294b4824", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(8344, 191)" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "\n", "df[\"Report_Date_as_YYYY-MM-DD\"] = pd.to_datetime(df[\"Report_Date_as_YYYY-MM-DD\"], format='%Y-%m-%d')\n", "df.set_index(\"Report_Date_as_YYYY-MM-DD\")\n", "df.shape" ] }, { "cell_type": "code", "execution_count": 5, "id": "4bd367af-8385-406a-87ee-5aa68b3a8b7a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "8345\n" ] } ], "source": [ "!cat COT_CFTC_2024.csv | wc -l" ] }, { "cell_type": "code", "execution_count": 6, "id": "fc275bed-93a3-4c66-87e0-7312aa712f6a", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", " | Market_and_Exchange_Names | \n", "As_of_Date_In_Form_YYMMDD | \n", "Report_Date_as_YYYY-MM-DD | \n", "CFTC_Contract_Market_Code | \n", "CFTC_Market_Code | \n", "CFTC_Region_Code | \n", "CFTC_Commodity_Code | \n", "Open_Interest_All | \n", "Prod_Merc_Positions_Long_All | \n", "Prod_Merc_Positions_Short_All | \n", "... | \n", "Conc_Net_LE_4_TDR_Long_Other | \n", "Conc_Net_LE_4_TDR_Short_Other | \n", "Conc_Net_LE_8_TDR_Long_Other | \n", "Conc_Net_LE_8_TDR_Short_Other | \n", "Contract_Units | \n", "CFTC_Contract_Market_Code_Quotes | \n", "CFTC_Market_Code_Quotes | \n", "CFTC_Commodity_Code_Quotes | \n", "CFTC_SubGroup_Code | \n", "FutOnly_or_Combined | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | \n", "WHEAT-SRW - CHICAGO BOARD OF TRADE | \n", "240813 | \n", "2024-08-13 | \n", "001602 | \n", "CBT | \n", "0 | \n", "1 | \n", "401604 | \n", "68899 | \n", "85867 | \n", "... | \n", "39.0 | \n", "31.7 | \n", "54.2 | \n", "42.6 | \n", "(CONTRACTS OF 5,000 BUSHELS) | \n", "001602 | \n", "CBT | \n", "1 | \n", "A10 | \n", "FutOnly | \n", "
1 | \n", "WHEAT-SRW - CHICAGO BOARD OF TRADE | \n", "240806 | \n", "2024-08-06 | \n", "001602 | \n", "CBT | \n", "0 | \n", "1 | \n", "431547 | \n", "73401 | \n", "85750 | \n", "... | \n", "41.6 | \n", "32.3 | \n", "57.3 | \n", "41.7 | \n", "(CONTRACTS OF 5,000 BUSHELS) | \n", "001602 | \n", "CBT | \n", "1 | \n", "A10 | \n", "FutOnly | \n", "
2 | \n", "WHEAT-SRW - CHICAGO BOARD OF TRADE | \n", "240730 | \n", "2024-07-30 | \n", "001602 | \n", "CBT | \n", "0 | \n", "1 | \n", "432863 | \n", "73086 | \n", "83875 | \n", "... | \n", "46.0 | \n", "28.5 | \n", "60.8 | \n", "38.4 | \n", "(CONTRACTS OF 5,000 BUSHELS) | \n", "001602 | \n", "CBT | \n", "1 | \n", "A10 | \n", "FutOnly | \n", "
3 | \n", "WHEAT-SRW - CHICAGO BOARD OF TRADE | \n", "240723 | \n", "2024-07-23 | \n", "001602 | \n", "CBT | \n", "0 | \n", "1 | \n", "421227 | \n", "63636 | \n", "81374 | \n", "... | \n", "48.3 | \n", "27.9 | \n", "63.5 | \n", "37.7 | \n", "(CONTRACTS OF 5,000 BUSHELS) | \n", "001602 | \n", "CBT | \n", "1 | \n", "A10 | \n", "FutOnly | \n", "
4 | \n", "WHEAT-SRW - CHICAGO BOARD OF TRADE | \n", "240716 | \n", "2024-07-16 | \n", "001602 | \n", "CBT | \n", "0 | \n", "1 | \n", "413885 | \n", "64297 | \n", "81558 | \n", "... | \n", "50.4 | \n", "28.0 | \n", "65.9 | \n", "37.3 | \n", "(CONTRACTS OF 5,000 BUSHELS) | \n", "001602 | \n", "CBT | \n", "1 | \n", "A10 | \n", "FutOnly | \n", "
5 rows × 191 columns
\n", "\n", " | Open_Interest_All | \n", "Pct_of_OI_Prod_Merc_Long_Old | \n", "Pct_of_OI_Prod_Merc_Short_Old | \n", "
---|---|---|---|
Report_Date_as_YYYY-MM-DD | \n", "\n", " | \n", " | \n", " |
2024-08-13 | \n", "147859 | \n", "3.1 | \n", "25.3 | \n", "
2024-08-06 | \n", "147537 | \n", "3.0 | \n", "25.4 | \n", "
2024-07-30 | \n", "151437 | \n", "2.7 | \n", "25.4 | \n", "
2024-07-23 | \n", "157106 | \n", "2.2 | \n", "25.8 | \n", "
2024-07-16 | \n", "166641 | \n", "1.7 | \n", "26.0 | \n", "