Posts

Showing posts with the label dataframe

R: How to retrieve a column name of a data frame

Image
Clash Royale CLAN TAG #URR8PPP R: How to retrieve a column name of a data frame I am trying to extract the colnames of a data frame, based on the values in the cells. My data is a series of a couple hundred categories, with a simple binary 0 or 1 in the cells to indicate which column name I want in my new df. To illustrate my point: year cat1 cat2 cat3 ... catN 2000 0 0 1 0 2001 1 0 0 0 2002 0 0 0 1 .... 2018 0 1 0 0 I am trying to get a df like: year category 2000 cat3 2001 cat1 2002 catN .... 2018 cat2 My code: newdf <- as.data.frame(colnames(mydf)[which(mydf == "1", arr.ind = TRUE)[2]]) But alas this only returns one category name! Any help would be greatly appreciated! 1 Answer 1 A possible solution is this: library(tidyverse) df = data.frame(year = 2000:2002, cat1 = c(0,0,1), cat2 = c(1...

Copy a selection and insert it to the bottom(n times) Excel

Image
Clash Royale CLAN TAG #URR8PPP Copy a selection and insert it to the bottom(n times) Excel Suppose I have a template a,a,a,a,a,a,a b,b,b,b,b,b,b c,c,c,c,c,c,c I want to repeat this block n times. let's say I want to repeat 3 times so n = 3. a,a,a,a,a,a,a b,b,b,b,b,b,b c,c,c,c,c,c,c a,a,a,a,a,a,a b,b,b,b,b,b,b c,c,c,c,c,c,c a,a,a,a,a,a,a b,b,b,b,b,b,b c,c,c,c,c,c,c OK, let's say that. Where are you stuck? – Roland 33 mins ago Look into For Loops – QHarr 31 mins ago Have you tried using record macro? – Solar Mike 25 mins ago ...

Multiply columns by columns using substrings

Image
Clash Royale CLAN TAG #URR8PPP Multiply columns by columns using substrings I'm relatively new to R and was struggling with potentially a very simple problem. I have data that has multiple columns named in a similar way. Here is a sample data: df = data.frame(PPID = 1:50, time1 = sample(c(0,1), 50, replace = TRUE), time2 = sample(c(0,1), 50, replace = TRUE), time3 = sample(c(0,1), 50, replace = TRUE), condition1 = sample(c(0:3), 50, replace = TRUE), condition2 = sample(c(0:3), 50, replace = TRUE)) In my actual data, I have much more columns - approximately 50 for time and 10 for condition. I want to multiply week columns and condition columns, e.g. in that sample data it should give me 6 extra columns, like: time1_condition1, time1_condition2, time2_condition1, time2_condition2, time3_condition1, time3_condition2. I tried solutions that were suggested in this thread but they did not work (presumably bec...

printing product headers that have 0 as a value in a separate header GUI

Image
Clash Royale CLAN TAG #URR8PPP printing product headers that have 0 as a value in a separate header GUI For my code, I have to find a certain product and the values associated with it for a specific date( that are headers in the csv file) and take those numbers and manipulate them. However, there are blank values for many of the products ( because those products are not produced for those specific months), thus the reason I produced my current code, because I did not want to display adjusted values for numbers that were either blank or 0. import pandas as pd import csv import numpy as np import os from tkinter import* master = Tk() fileVar = StringVar() fileLabel = Label(master, textvariable=fileVar, font=('Consolas', 9)) fileLabel.grid(row=8, column=1) fileVar2 = StringVar() fileLabel2 = Label(master, textvariable=fileVar2, font=('Consolas', 9)) fileLabel2.grid(row=8, column=2) readfile = pd.read_csv('50.csv') filevalues= readfile.loc[readfile['Custome...

Anti-Join Pandas

Image
Clash Royale CLAN TAG #URR8PPP Anti-Join Pandas I have two tables and I would like to append them so that only all the data in table A is retained and data from table B is only added if its key is unique (Key values are unique in table A and B however in some cases a Key will occur in both table A and B). I think the way to do this will involve some sort of filtering join (anti-join) to get values in table B that do not occur in table A then append the two tables. I am familiar with R and this is the code I would use to do this in R. library("dplyr") ## Filtering join to remove values already in "TableA" from "TableB" FilteredTableB <- anti_join(TableB,TableA, by = "Key") ## Append "FilteredTableB" to "TableA" CombinedTable <- bind_rows(TableA,FilteredTableB) How would I achieve this in python? By key do you mean row index, column index, or cell? – Jossie Calderon ...

Split mixed type DataFrame into two columns?

Image
Clash Royale CLAN TAG #URR8PPP Split mixed type DataFrame into two columns? I'm munging a report I loaded into a DataFrame. The report's SKU column has mixed datatypes. I want to split the column into two new columns (SUBTOTAL and SKU) based on cell data type (str, int). Following the example from a similar question I get a boolean column. Ok df['SUBTOTAL'] = df['SKU'].apply(lambda x: isinstance(x, str)) SKU AMOUNT SUBTOTAL 7 4410 1 False 8 4200 5 False 9 total 6 True 11 4250 0 False 12 4255 0 False I'm doing this in a Jupyter Notebook. Here's the thing that's driving me crazy. If I first call the above line, and wrap the code with df , and rerun that cell, I get what I want. df df['SUBTOTAL'] = df[df['SKU'].apply(lambda x: isinstance(x, str))] ...

How to fill a particular value with mean value of the column between first row and the corresponding row in pandas dataframe

Image
Clash Royale CLAN TAG #URR8PPP How to fill a particular value with mean value of the column between first row and the corresponding row in pandas dataframe I have a df like this, A B C D E 1 2 3 0 2 2 0 7 1 1 3 4 0 3 0 0 0 3 4 3 I am trying to replace all the 0 with mean() value between the first row and the 0 value row for the corresponding column, My expected output is, A B C D E 1.0 2.00 3.000000 0.0 2.0 2.0 1.00 7.000000 1.0 1.0 3.0 4.00 3.333333 3.0 1.0 1.5 1.75 3.000000 4.0 3.0 1 Answer 1 IIUC def f(x): for z in range(x.size): if x[z] == 0: x[z] = np.mean(x[:z+1]) return x df.astype(float).apply(f) A B C D E 0 1.0 2.00 3.000000 0.0 2.0 1 2.0 1.00 7.000000 1.0 1.0 2 3.0 4.00 3.333333 3.0 1.0 3 1.5 1.75 3....

Transposing specific duplicated column data into rows given ID #

Image
Clash Royale CLAN TAG #URR8PPP Transposing specific duplicated column data into rows given ID # I have a data frame that consists of ID #'s in the first column and multiple transactions for that given ID. For example: ID Transaction 1111 $13 1111 $55 1111 $4 1112 $27 1112 $40 1113 $12 1114 $100 1114 $60 1114 $55 What I am trying to do is to have only one line of the Customer ID and then creating multiple transactions for each instance. For example: ID Transaction1 Transaction2 Transaction3 1111 $13 $55 $4 1112 $27 $40 1113 $12 1114 $100 $60 $55 Any help would be appreciated. I've been trying to use for loops and what not but I keep getting lost in my work and have been looking for an easier way to do this. Possible duplicate of How to reshape data from long to wide format? – Rui Barradas 5 mins ago ...

Loop to identify data location in a Dataframe, and populate to new Dataframe (R)

Image
Clash Royale CLAN TAG #URR8PPP Loop to identify data location in a Dataframe, and populate to new Dataframe (R) So I have a dataframe of user IDs as columns with rows (corresponding to period intervals) with binary variables (where an event happened) like this: date Id1 Id2 id3 id4 id5 row1 1 0 0 1 0 row2 0 0 1 1 0 row3 0 1 0 0 1 row4 1 1 0 0 0 row5 0 0 1 1 1 ... I am trying to build a loop that runs through each row of each column which identifies any cell with a 1 and populates a new data frame with the row number of each occurrence, e.g: occ. Id1 Id2 id3 id4 id5 1 1 3 2 1 3 2 4 4 5 2 5 3 5 4 5 I am pretty lost in how to approach this, if anyone is able to help? Your second frame isn't really a frame since you have different lengths of each column. Additionally, your sample frames don't match, can you edit your question to make them representative?...

How to plot column from a dataframe

Image
Clash Royale CLAN TAG #URR8PPP How to plot column from a dataframe I am trying to plot integer columns of dataframe. I am trying by following way for i in df: if df[i].dtypes == 'int64': df[i].plot.kde() But it is plotting all in the same graph. I am new to it and would like to know how can I do it? This answer may help your question :https://stackoverflow.com/questions/22483588/how-can-i-plot-separate-pandas-dataframes-as-subplots df.select_dtypes(include=["integer"]).columns.size calculates your axis size. – Ceyhun yesterday df.select_dtypes(include=["integer"]).columns.size DO you want one graph per time, subplots?.. – Joe yesterday ...

Subtle issue with joining/merging dataframes with the same column names

Image
Clash Royale CLAN TAG #URR8PPP Subtle issue with joining/merging dataframes with the same column names So I have a main set of data that looks like this: value_num code value_letter 1 CDX A 2 DEF B 3 RPQ C 4 EEE D 5 FFX E 6 TRE F And two other tables which we'll call map1 and map2 song album_code song_code Song1 CDX GIB Song2 DEF FRE Song3 RPQ SSS song album_code song_code Song4 REA EEE Song5 VEY FFX Song6 LFM TRE I want to join the main table with map1 where album_code is joined on code. Then I want to join map2 on this new table where song_code is joined on code. Ideally the final result looks like this: value_num code value_letter song album_code song_code 1 CDX A Song1 CDX GIB 2 DEF B Song2 DEF FRE 3 RPQ C Son...

Exported and imported DataFrames differ but should be the same

Image
Clash Royale CLAN TAG #URR8PPP Exported and imported DataFrames differ but should be the same I tried to import some data from an Excel file to a pandas DataFrame, convert it into a csv file and read it back in (need to do some further file based handling on that exported csv file later on, so that is a necessary step). For the sake of data integrity, exported and re-imported data should be the same. So, I compared the different DataFrames and encountered, that these are not the same, at least according to pandas' .equals() function. .equals() I thought this might be an issue related to string encoding when exporting and re-importing the data since I had to transfer char encoding etc. while file handling. However, I was able to reproduce similar behavior without any encoding-related issues as follows: import pandas as pd import numpy as np # https://stackoverflow.com/a/32752318 df1 = pd.DataFrame(np.random.randint(0, 10, size=(10, 4)), columns=list('ABCD')) df1.to_csv(...

Pandas column creation methods

Pandas column creation methods There are many methods for creating new columns in Pandas (I may have missed some in my examples so please let me know if there are others and I will include here) and I wanted to figure out when is the best time to use each method. Obviously some methods are better in certain situations compared to others but I want to evaluate it from a holistic view looking at efficiency , readability , and usefulness . I'm primarily concerned with the first three but included other ways simply to show it's possible with different approaches. Here's your sample dataframe : dataframe df = pd.DataFrame({'a':[1,2,3],'b':[4,5,6]}) Most commonly known way is to name a new column such as df['c'] and use apply : df['c'] apply df['c'] = df['a'].apply(lambda x: x * 2) df a b c 0 1 4 2 1 2 5 4 2 3 6 6 Using assign can accomplish the same thing: assign df = df.assign(c = lambda x: x['a'] * 2) df ...