Ciugk

Clash Royale CLAN TAG #URR8PPP Space delimiter in CSV import to Python I know there are more than a few questions regarding space delimiters in CSV files. I have a CSV file that appears to be separated by a space. When importing to Python, I have tried every code out there to identify space as a delimiter. However, I keep getting error messages. For example: test_filepath = 'test_data.csv' with codecs.open(test_filepath, "r", "Shift-JIS", "ignore") as file: # import UTF8 based csv file test_df = pd.read_table( file, delim_whitespace=True ) this yields the following error: EmptyDataError: No columns to parse from file when I try this: test_filepath = 'test_data.csv' with codecs.open(test_filepath, "r", "Shift-JIS", "ignore") as file: # import UTF8 based csv file test_df = pd.read_table( file, delimiter=" " ) it gives the same error. when i try this: test_filepath = 'test_data....

Clash Royale CLAN TAG #URR8PPP Pandas how to calculate bygroup result based on the length of the each group and a count value of another column I want to calculate the scoring rate of each zone by using bygroup in pandas, but not sure how to do it: Suppose the df has two columns as: Shot_type Shot_zone Goal Penalty_area Saved Penalty_area Goal Goal Box Saved Goal Box Here I want to groupy by Shot_zone, and calculate the scoring rate based on Shot_type's Goal counts / len() of each type Shot_zone. Here each Shot_zone has 1 goal and 1 saved, so the result should be like: Penalty_area 50% Goal Box 50% Is there any understandable approach to do so using Pandas? Thank you very much! 3 Answers 3 Using pd.crosstab(df.Shot_type,df.Shot_zone,normalize='index') Out[662]: Shot_zone GoalBox Penalty_area Shot_type Goal 0.5 ...

Clash Royale CLAN TAG #URR8PPP Python & Pandas: How to query if a list-type column contains something? I have a dataframe, which contains info about movies. It has a column called genre , which contains a list of genres it belongs to. For example genre df['genre'] ## returns 0 ['comedy', 'sci-fi'] 1 ['action', 'romance', 'comedy'] 2 ['documentary'] 3 ['crime','horror'] ... I want to know how can I query the df, so it returns the movie belongs to a cerain genre? For example, something may like df['genre'].contains('comedy') returns 0, 1. df['genre'].contains('comedy') I know for a list, I can do things like 'comedy' in ['comedy', 'sci-fi'] but in pandas, I didn't find something similar, the only thing I know is df['genre'].str.contains() , but it didn't work for the list type. df['genre'].str.contains() ...

Clash Royale CLAN TAG #URR8PPP Detecting CSV file truncation when NA is a valid value If I use the Pandas read_csv() function, elements of short rows are mapped to NaN by default. I would like to suppress that mapping while interpreting NA as NaN . I'm mostly interested in file truncation as a result of transmission problems, but short rows in the middle of the file should feature the same missing" value. I tried messing around with na_filter=False and keep_default_na=False , and while each seemed to map empty cells to the empty string, neither mapped the string NA to NaN . NA NaN missing" na_filter=False keep_default_na=False NA NaN Is there a way to have my cake ( NA => NaN ) and eat it too (missing values not mapped to NaN )? (I'm using Pandas 0.22.0 w/ Python 3.6. ) NA NaN NaN Pandas 0.22.0 w/ Python 3.6. Example: col1,col2,col3,col4 1,2,NA,4 4,5 12 Assume the file has been truncated, so the characters "12" are the last in the file (no EOF). Wit...

Clash Royale CLAN TAG #URR8PPP Create new columns from aggregated categories 2 SK_ID_CURR CREDIT_ACTIVE CREDIT_TYPE 0 215354 Closed Consumer credit 1 215354 Active Credit card 2 215354 Active Consumer credit 3 215354 Active Credit card 4 215354 Active Consumer credit 5 215354 Active Credit card 6 215354 Active Consumer credit 7 162297 Closed Consumer credit 8 162297 Closed Consumer credit 9 162297 Active Credit card 10 162297 Active Credit card 11 162297 Closed Consumer credit 12 162297 Active Mortgage 13 402440 Active Consumer credit 14 238881 Closed Credit card I have the table above. I'd like to aggregate each column per id. For example I need to count the number of active and closed credits per SK_ID_CURR , and then make a column for active_credits and closed_credits, with the counted values. And the same for CREDIT_TYPE . SK_ID_CURR CREDIT_TYPE like: SK_ID_CURR CREDIT_ACTIVE CREDIT_CLOSED CONSUMER_CREDIT CREDIT_CARD 215...

Clash Royale CLAN TAG #URR8PPP Create new columns from aggregated categories I have a dataframe looks like: SK_ID_CURR CREDIT_ACTIVE 0 215354 Closed 1 215354 Active 2 215354 Active 3 215354 Active 4 215354 Active 5 215354 Active 6 215354 Active 7 162297 Closed 8 162297 Closed 9 162297 Active I would like to aggregate the number of active and closed credits for each id, and then make a new column for Active_credits , Closed_credits with the number of corresponding active and closed credits for each id. Active_credits Closed_credits What is your expected output? – Akshay Nevrekar 10 mins ago 2 Answers 2 You can use pandas.crosstab , which avoids your suggested intermediary step: pandas.crosstab res = pd.crosstab(df['...

Clash Royale CLAN TAG #URR8PPP Perform cumulative count on a pandas column taking order into account I have a pandas data frame pd.DataFrame(columns=["A", "B"], data=[['id1','a'],['id1','a'], ['id1','a'], ['id1','b'], ['id1','b'], ['id1','a'], ['id1','a'], ['id2','c'], ['id2','c'], ['id2','a'], ['id2','c']]) A B 0 id1 a 1 id1 a 2 id1 a 3 id1 b 4 id1 b 5 id1 a 6 id1 a 7 id2 c 8 id2 c 9 id2 a 10 id2 c I want to do a group by that gives me cumulative sum of occurrences in B column considering the order A B C 0 id1 a 3 1 b 2 2 a 2 3 id2 c 2 4 a 1 5 c 1 1 Answer 1 You can use itertools.groupby followed by a list comprehension. This wor...

Clash Royale CLAN TAG #URR8PPP PandasDataframe to_datetime error I have got the following indices on a dataframe: data extracted Index(['2014-06-30 00:00:00.0', '2014-07-07 00:00:00.0', '2014-08-11 00:00:00.0', '2014-08-18 00:00:00.0', '2014-08-25 00:00:00.0', '2014-09-08 00:00:00.0', '2014-09-22 00:00:00.0', '2014-09-29 00:00:00.0', '2014-10-06 00:00:00.0', '2014-10-27 00:00:00.0', '2014-11-24 00:00:00.0', '2014-12-15 00:00:00.0', '2014-12-29 00:00:00.0', '2015-01-05 00:00:00.0', '2015-01-19 00:00:00.0', '2015-01-26 00:00:00.0', '2015-02-02 00:00:00.0', '2015-02-16 00:00:00.0', '2015-02-23 00:00:00.0', '2015-04-13 00:00:00.0', '2015-04-20 00:00:00.0', '2015-05-04 00:00:00.0', '2015-05-25 00:00:00.0', '2015-06-01 00:00:00.0...

Clash Royale CLAN TAG #URR8PPP Logistic Regression (tried to implement) but keep getting too accurate model So I've tried to follow the below tutorial and keep getting a 99% accurate algorithm, but I seriously doubt this. i'm trying to predict risk of injury which is highlighted by IntTot below. https://towardsdatascience.com/building-a-logistic-regression-in-python-step-by-step-becd4d56c9c8 Input: Note the column headings are not correct below, first column is index. TotClear BLD_HEIGHT INITIAL_CALL_HOUR TOTAL_NUM_PERSONNEL InjTot TotClearS TOTAL_NUM_PERSONNELS year ALARM_TO_FD 141677 8.316667 0 17 14 0.0 (7.333, 9.683] (7.0, 594.0] 2011 05 314976 21.483333 0 9 4 0.0 (21.0, 28.45] (-0.001, 4.0] 2013 03 215834 5.666667 0 23 4 0.0 NaN NaN 2012 03 318900 13.966667 0 23 20 0.0 (11.867, 14.167] (7.0, 594.0] 2013 01 468452 4.050000 0 5 4 0.0 (-58.834, 7.333] (-0.001...

Clash Royale CLAN TAG #URR8PPP Pandas Multiindex Series Processing I have a pandas 2 index serie that I got from a .groupby() on my original DataFrame: label ncsc False 0 297 1 537 2 333 3 207 4 51 5 12 6 4 7 2 True 0 29 1 68 2 35 3 29 4 35 5 18 6 8 7 2 Name: ncsc, dtype: int64 I would like to be able to compute for each 'ncsc' the True rate, ie, for 'ncsc'=6, True rate = 8/(4+8) = 0.66. Do you see a way to do that with =out using a loop but with a 'pandas synthax'?? Thanks 1 Answer 1 Convert the wanted index level to a column and group by that column: df.reset_index(level=1).groupby('ncsc').mean() By clicking ...

Clash Royale CLAN TAG #URR8PPP How to convert a pandas dataframe from a string based categorical column to a numeric representation I have a column in a dataframe which looks like this: df['label'] ['some_label', 'some_label', 'a_diff_label', 'a_diff_label',...] I want it to convert it to something like this: [1,1,0,0,...] 2 Answers 2 Since the similar question I found was very complex and hard to understand, I am posting a simple answer. Just do this: df['label'] = (df['label'] == 'some_label').astype(int) There are lot of ways to achieve this (etc, factor) pd.Series(['some_label', 'some_label', 'a_diff_label', 'a_diff_label']).astype('category').cat.codes Out[19]: 0 1 1 1 2 0 3 0 dtype: int8 By clicking "Post Your Answer", you ack...

Clash Royale CLAN TAG #URR8PPP printing product headers that have 0 as a value in a separate header GUI For my code, I have to find a certain product and the values associated with it for a specific date( that are headers in the csv file) and take those numbers and manipulate them. However, there are blank values for many of the products ( because those products are not produced for those specific months), thus the reason I produced my current code, because I did not want to display adjusted values for numbers that were either blank or 0. import pandas as pd import csv import numpy as np import os from tkinter import* master = Tk() fileVar = StringVar() fileLabel = Label(master, textvariable=fileVar, font=('Consolas', 9)) fileLabel.grid(row=8, column=1) fileVar2 = StringVar() fileLabel2 = Label(master, textvariable=fileVar2, font=('Consolas', 9)) fileLabel2.grid(row=8, column=2) readfile = pd.read_csv('50.csv') filevalues= readfile.loc[readfile['Custome...

Clash Royale CLAN TAG #URR8PPP Anti-Join Pandas I have two tables and I would like to append them so that only all the data in table A is retained and data from table B is only added if its key is unique (Key values are unique in table A and B however in some cases a Key will occur in both table A and B). I think the way to do this will involve some sort of filtering join (anti-join) to get values in table B that do not occur in table A then append the two tables. I am familiar with R and this is the code I would use to do this in R. library("dplyr") ## Filtering join to remove values already in "TableA" from "TableB" FilteredTableB <- anti_join(TableB,TableA, by = "Key") ## Append "FilteredTableB" to "TableA" CombinedTable <- bind_rows(TableA,FilteredTableB) How would I achieve this in python? By key do you mean row index, column index, or cell? – Jossie Calderon ...

Clash Royale CLAN TAG #URR8PPP Split mixed type DataFrame into two columns? I'm munging a report I loaded into a DataFrame. The report's SKU column has mixed datatypes. I want to split the column into two new columns (SUBTOTAL and SKU) based on cell data type (str, int). Following the example from a similar question I get a boolean column. Ok df['SUBTOTAL'] = df['SKU'].apply(lambda x: isinstance(x, str)) SKU AMOUNT SUBTOTAL 7 4410 1 False 8 4200 5 False 9 total 6 True 11 4250 0 False 12 4255 0 False I'm doing this in a Jupyter Notebook. Here's the thing that's driving me crazy. If I first call the above line, and wrap the code with df , and rerun that cell, I get what I want. df df['SUBTOTAL'] = df[df['SKU'].apply(lambda x: isinstance(x, str))] ...

Clash Royale CLAN TAG #URR8PPP How to square the individual matrix value using python? I am trying to implement the Cost function in python. Assume my data have X (loading from txt file) and my theta value is [[0] [0]] For that I have implemented as below: import pandas as pd import numpy as np import matplotlib.pyplot as plt from scipy.linalg import fractional_matrix_power load_data = pd.read_csv('C:python_programex1data1.txt',sep = ",",header = None) feature_vale = load_data[0] y = np.matrix(load_data[1]) m = len(feature_vale) #print(m) #plt.scatter(load_data[0],load_data[1]) df = pd.DataFrame(pd.Series(1,index= range(0,m))) df[1] = load_data[0] X = np.matrix(df) row_theta = np.zeros(2,dtype = int) theta = np.array([row_theta]) # Transpose the array print(theta.T) prediction = np.matmul(X,theta.T) error = (prediction-y) print(error) Output of the error I got as expect...