Replacing Pandas or Numpy Nan with a None to use with MysqlDB


Question

I am trying to write a Pandas dataframe (or can use a numpy array) to a mysql database using MysqlDB . MysqlDB doesn't seem understand 'nan' and my database throws out an error saying nan is not in the field list. I need to find a way to convert the 'nan' into a NoneType.

Any ideas?

1
86
1/4/2019 12:20:22 PM

Accepted Answer

@bogatron has it right, you can use where, it's worth noting that you can do this natively in pandas:

df1 = df.where((pd.notnull(df)), None)

Note: this changes the dtype of all columns to object.

Example:

In [1]: df = pd.DataFrame([1, np.nan])

In [2]: df
Out[2]: 
    0
0   1
1 NaN

In [3]: df1 = df.where((pd.notnull(df)), None)

In [4]: df1
Out[4]: 
      0
0     1
1  None

Note: what you cannot do recast the DataFrames dtype to allow all datatypes types, using astype, and then the DataFrame fillna method:

df1 = df.astype(object).replace(np.nan, 'None')

Unfortunately neither this, nor using replace, works with None see this (closed) issue.


As an aside, it's worth noting that for most use cases you don't need to replace NaN with None, see this question about the difference between NaN and None in pandas.

However, in this specific case it seems you do (at least at the time of this answer).

143
8/27/2019 12:20:55 PM

df = df.replace({pd.np.nan: None})

Credit goes to this guy here on Github issue.


Licensed under: CC-BY-SA with attribution
Not affiliated with: Stack Overflow
Icon