Gtjkyu

How to manage Numpy arrays in Pandas DataFrames Let's assume one has a DataFrame with some integers values and some arrays defined somehow: df = pd.DataFrame(np.random.randint(0,100,size=(5, 1)), columns=['rand_int']) array_a = np.arange(5) array_b = np.arange(7) df['array_a'] = df['rand_int'].apply(lambda x: array_a[:x]) df['array_b'] = df['rand_int'].apply(lambda x: array_b[:x]) Some questions which can help me understand how to manage Numpy arrays with Pandas DataFrames: array_diff So you want to multiply both array a and array b by the corresponding value in rand_int? – user3483203 Jul 2 at 7:20 Not only that, but also define another column in df which is the np.setdiff1d between the rows in array_a and array_b . Thank you – espogian ...

Saving a Numpy array as an image I have a matrix in the type of a Numpy array. How would I write it to disk it as an image? Any format works (png, jpeg, bmp...). One important constraint is that PIL is not present. I'd just like to note that some of the answers below, and surely some of the people coming and finding this question, do not meet the constraint listed above of being without PIL. Since some askers and some answers both avoid that constraint, I encourage anyone who's here and doesn't mind having PIL to look below, and any non-PIL answers (new or old) to mention that they're a PIL-is-used type of answer, to distinguish themselves from answers meeting the original constraint. – lindes Oct 30 '13 at 15:46 14 Answers 14 Yo...

How to compute Pairwise L1 Distance matrix on very large images in neighborhood only? I am working on Deep learning approach for my project. And I need to calculate Distance Matrix on 4D Tensor which will be of size N x 128 x 64 x 64 (Batch Size x Channels x Height x Width) . The distance matrix for this type of tensor will of size N x 128 x 4096 x 4096 and it will be impossible to fit this type of tensor in GPU, even on CPU it will require lot of memory. So, I would like to calculate the distance matrix only in some neighborhood pixels (say within radius of 5) and consider this rectangular matrix for further processing in neural network. With this approach my distance matrix will be of size N x 128 x 4096 x 61 . It will take less memory in comparison to full distance matrix. Precisely, I am trying to implement the Convolution Random Walk Networks for Semantic Segmentation. This network needs to calculate the Pairwise L1 Distance for features. Architecture Just to add this type of Di...

Numpy […,None] I have found myself needing to add features to existing numpy arrays which has led to a question around what the last portion of the following code is actually doing: np.ones(shape=feature_set.shape)[...,None] Set-up As an example, let's say I wish to solve for linear regression parameter estimates by using numpy and solving: Assume I have a feature set shape (50,1), a target variable of shape (50,), and I wish to use the shape of my target variable to add a column for intercept values. It would look something like this: # Create random target & feature set y_train = np.random.randint(0,100, size = (50,)) feature_set = np.random.randint(0,100,size=(50,1)) # Build a set of 1s after shape of target variable int_train = np.ones(shape=y_train.shape)[...,None] # Able to then add int_train to feature set X = np.concatenate((int_train, feature_set),1) What I Think I Know I see the difference in output when I include [...,None] vs when I leave it off. Here it is: The ...

Why does this order of the Gaussian filter in scipy give the x and y derivative? I'm using a Gaussian filter with Scipy and I saw this code online which I'm curious about. imx = zeros(im.shape) filters.gaussian_filter(im, (sigma,sigma), (0,1), imx) imy = zeros(im.shape) filters.gaussian_filter(im, (sigma,sigma), (1,0), imy) For the first Gaussian filter call, the order is (0,1) and according to this link, that should give the the first order derivative of a Gaussian in y-direction. However, on running the code, I can see that the Gaussian is along the X direction. The same thing applies to imy. Why does the code work that way? For reference, running: filters.gaussian_filter(im, (sigma, sigma), (0, 1), output= imx) on this array: [[0 3 2] [1 4 1] [3 4 2]] Returns: [[0.00071801 0.00148952 0.00077151] [0.0006947 0.00144284 0.00074815] [0.00067141 0.00139622 0.00072482]] Which is a Gaussian in the x direction, even though the order (0, 1) suggests that it should be in the y dire...

How do I compute derivative using Numpy? How do I calculate the derivative of a function, for example y = x 2 +1 using numpy ? numpy Let's say, I want the value of derivative at x = 5... You need to use Sympy: sympy.org/en/index.html Numpy is a numeric computation library for Python – prrao Mar 26 '12 at 16:55 Alternatively, do you want a method for estimating the numerical value of the derivative? For this you can use a finite difference method, but bear in mind they tend to be horribly noisy. – Henry Gomersall Mar 26 '12 at 17:11 7 Answers 7 You have four options Finite differences require no ex...

Importing PNG files into Numpy? I have about 200 grayscale PNG images stored within a directory like this. 1.png 2.png 3.png ... ... 200.png I want to import all the PNG images into Numpy and then later want to apply k-means to generate a dictionary of patches using k-means (scikit) Does anybody know a python library that could load these images into numpy on a fly? 5 Answers 5 Using just scipy, glob and having PIL installed ( pip install pillow ) you can use scipy's imread method: pip install pillow from scipy import misc import glob for image_path in glob.glob("/home/adam/*.png"): image = misc.imread(image_path) print image.shape print image.dtype UPDATE According to the doc, scipy.misc.imread is deprecated starting SciPy 1.0.0, and will be removed in 1.2.0. Consider using imageio.imread instead . See the answer by Charles. scipy.misc.imread imageio.imread instead ...

Working with TIFFs (import, export) in Python using numpy I need a python routine that can open and import TIFF images into numpy arrays, so I can analyze and modify the contained data and afterwards save them as TIFFs again. (They are basically light intensity maps in greyscale, representing the respective values per pixel) I tried to find something, but there is no documentation on PIL methods concerning TIFF. I tried to figure it out, but only got bad mode/ file type not supported errors. What do I need to use here? 6 Answers 6 First, I downloaded a test TIFF image from this page called a_image.tif . Then I opened with PIL like this: a_image.tif >>> from PIL import Image >>> im = Image.open('a_image.tif') >>> im.show() This showed the rainbow image. To convert to a numpy array, it's as simple as: >>> import numpy >>> imarray = numpy.array(i...