How to Handle SettingWithCopyWarning in Pandas?

General Discussions
#1

Background: After upgrading Pandas from version 0.11 to 0.13.0rc1, my application started generating new warnings, such as:

E:\FinReporter\FM_EXT.py:449: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
  quote_df['TVol'] = quote_df['TVol'] / TVOL_SCALE

I am attempting to set a value on a copy of a slice from a DataFrame, which might not have the intended effect. This can lead to unintended consequences and bugs in my code.

Example Function Triggering the Warning

Here’s the function that generates the warning:

def _decode_stock_quote(list_of_150_stk_str):
    """Decode the webpage and return a DataFrame."""

    from cStringIO import StringIO

    str_of_all = "".join(list_of_150_stk_str)
    quote_df = pd.read_csv(StringIO(str_of_all), sep=',', names=list('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefg'))
    quote_df.rename(columns={'A': 'STK', 'B': 'TOpen', 'C': 'TPCLOSE', 'D': 'TPrice', 'E': 'THigh', 'F': 'TLow', 'I': 'TVol', 'J': 'TAmt', 'e': 'TDate', 'f': 'TTime'}, inplace=True)
    quote_df = quote_df.ix[:, [0, 3, 2, 1, 4, 5, 8, 9, 30, 31]]
    quote_df['TClose'] = quote_df['TPrice']
    quote_df['RT'] = 100 * (quote_df['TPrice'] / quote_df['TPCLOSE'] - 1)
    quote_df['TVol'] = quote_df['TVol'] / TVOL_SCALE
    quote_df['TAmt'] = quote_df['TAmt'] / TAMT_SCALE
    quote_df['STK_ID'] = quote_df['STK'].str.slice(13, 19)
    quote_df['STK_Name'] = quote_df['STK'].str.slice(21, 30)
    quote_df['TDate'] = quote_df.TDate.map(lambda x: x[0:4] + x[5:7] + x[8:10])
    
    return quote_df

More Warning Messages

E:\FinReporter\FM_EXT.py:449: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
  quote_df['TVol'] = quote_df['TVol'] / TVOL_SCALE
E:\FinReporter\FM_EXT.py:450: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
  quote_df['TAmt'] = quote_df['TAmt'] / TAMT_SCALE
E:\FinReporter\FM_EXT.py:453: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
  quote_df['TDate'] = quote_df.TDate.map(lambda x: x[0:4] + x[5:7] + x[8:10])
#2

The SettingWithCopyWarning in Pandas was introduced to identify potentially confusing “chained” assignments, where the first selection returns a copy, leading to unexpected behavior. For example:

df[df['A'] > 2]['B'] = new_val  # new_val not set in df

To avoid this warning, it’s recommended to use .loc for assignment:

df.loc[df['A'] > 2, 'B'] = new_val

However, in your case, where you are overwriting the reference to the DataFrame, this pattern cannot be differentiated from the problematic chained assignment. Hence, the warning is a false positive in your scenario.

If you want to suppress this warning, you can do so by setting the Pandas option:

import pandas as pd
pd.options.mode.chained_assignment = None  # default='warn'

This will disable the warning for chained assignments.

#3

The purpose of the SettingWithCopyWarning in Pandas is to alert users, especially new users, that they may be working on a copy of the data rather than the original DataFrame. While there are false positives, where the warning is unnecessary, it’s meant to encourage safer coding practices.

If you encounter this warning and are sure about your code, you can turn off the warning:

import pandas as pd
pd.options.mode.chained_assignment = None  # default='warn'

Another approach is to set the is_copy flag to False for the DataFrame slice, effectively disabling the warning for that object:

dfa.is_copy = False

If you explicitly copy the DataFrame, no further warnings will occur:

dfa = df.ix[:, [1, 0]].copy()

While the code in the original post is technically a case for the warning, it’s not necessarily a false positive. Another way to avoid the warning is to perform the selection operation using reindex:

quote_df = quote_df.reindex(columns=['STK', ...])

Or, with a newer version of Pandas:

quote_df = quote_df.reindex(['STK', ...], axis=1)  # v.0.21
#4

To handle the SettingWithCopyWarning in Pandas, make a copy using .copy(deep=False) after slicing to avoid the warning:

df_copy = df[df['x'] > 2].copy(deep=False)
df_copy['foo'] = 'bar'

This ensures that the DataFrame is not marked as a copy, preventing the warning.