Ciugk

Clash Royale CLAN TAG #URR8PPP Faster Way to Generate Rolling Calculations on a list of columns within a groupby object I created this function to calculate the rolling stats for a list of feats in my df. This function works as intended but takes roughly 20min to run on my df which has about 1 million rows. Is there a faster way to do this in python/pandas ? def add_rolling_vars(df, feats, amounts, group): #creates rolling stats for a list of feats(columns) over a list of amounts[12,48](window sizes) #grouped by a group like $gvkey or $sector orig_feats = feats.copy() new_feats= for amount in amounts: for name in feats: df[group+'_'+name+f'_{amount}_sma'] = df.groupby(group)[name].rolling(amount,1).mean().values df[group+'_'+name+f'_{amount}_std'] = df.groupby(group)[name].rolling(amount,1).std().values df[group+'_'+name+f'_{amount}_min'] = df.groupby(group)[name].rolling(amount,1).min().values df[group...

Clash Royale CLAN TAG #URR8PPP why scipy.spatial.ckdtree runs slower than scipy.spatial.kdtree Normally,scipy.spatial.ckdtree runs much faster than scipy.spatial.kdtree. But in my case,scipy.spatial.ckdtree runs slower than scipy.spatial.kdtree. My code is as follows: import numpy as np from laspy.file import File from scipy import spatial from timeit import default_timer as timer inFile = File("Toronto_Strip_01.las") dataset = np.vstack([inFile.x, inFile.y, inFile.z]).transpose() print(dataset.shape) start=timer() tree = spatial.cKDTree(dataset) # balanced_tree = False end=timer() distance,index=tree.query(dataset[100,:],k=5) print(distance,index) print(end-start) start=timer() tree = spatial.KDTree(dataset) end=timer() dis,indices= tree.query(dataset[100,:],k=5) print(dis,indices) print(end-start) dataset.shape is (2727891, 3),dataset.max() is 4834229.32 But, in a test case, scipy.spatial.ckdtree runs much faster than scipy.spatial.kdtree,the code is as follows: import nump...

Clash Royale CLAN TAG #URR8PPP Cython: size attribute of memoryviews I'm using a lot of 3D memoryviews in Cython, e.g. cython.declare(a='double[:, :, ::1]') a = np.empty((10, 20, 30), dtype='double') I often want to loop over all elements of a . I can do this using a triple loop like a for i in range(a.shape[0]): for j in range(a.shape[1]): for k in range(a.shape[2]): a[i, j, k] = ... If I do not care about the indices i , j and k , it is more efficient to do a flat loop, like i j k cython.declare(a_ptr='double*') a_ptr = cython.address(a[0, 0, 0]) for i in range(size): a_ptr[i] = ... Here I need to know the number of elements ( size ) in the array. This is given by the product of the elements in the shape attribute, i.e. size = a.shape[0]*a.shape[1]*a.shape[2] , or more generally size = np.prod(np.asarray(a).shape) . I find both of these ugly to write, and the (albeit small) computational overhead bothers me. The nice way to do...